Make TC_LOG_MMAP (and TC_LOG_DUMMY) derive directly from TC_LOG, avoiding the
inheritance hierarchy TC_LOG_queued->TC_LOG_unordered.
Put the wakeup facility for commit_ordered() calls into the THD class.
Some renaming to get better names.
In sql/log.c, member function wait_for_update_bin_log, a condition is entered with
THD::enter_cond but is not exited. This might leave dangling references to the
mutex/condition in the per-thread information area.
To fix the problem, we call exit_cond to properly remove references to the mutex,
LOCK_log.
A CREATE...SELECT that fails is written to the binary log if a non-transactional
statement is updated. If the logging format is ROW, the CREATE statement and the
changes are written to the binary log as distinct events and by consequence the
created table is not rolled back in the slave.
In this patch, we opted to let the slave goes out of sync by not writting to the
binary log the CREATE statement. We do this by simply reseting the binary log's
cache.
mysql-test/suite/rpl/r/rpl_drop.result:
Added a test case.
mysql-test/suite/rpl/t/rpl_drop.test:
Added a test case.
sql/log.cc:
Introduced a function to clean up the cache.
sql/log.h:
Introduced a function to clean up the cache.
sql/sql_insert.cc:
Cleaned up the binary log cache if a CREATE...SELECT fails.
BUG#54872 MBR: replication failure caused by using tmp table inside transaction
Changed criteria to classify a statement as unsafe in order to reduce the
number of spurious warnings. So a statement is classified as unsafe when
there is on-going transaction at any point of the execution if:
1. The mixed statement is about to update a transactional table and
a non-transactional table.
2. The mixed statement is about to update a temporary transactional
table and a non-transactional table.
3. The mixed statement is about to update a transactional table and
read from a non-transactional table.
4. The mixed statement is about to update a temporary transactional
table and read from a non-transactional table.
5. The mixed statement is about to update a non-transactional table
and read from a transactional table when the isolation level is
lower than repeatable read.
After updating a transactional table if:
6. The mixed statement is about to update a non-transactional table
and read from a temporary transactional table.
7. The mixed statement is about to update a non-transactional table
and read from a temporary transactional table.
8. The mixed statement is about to update a non-transactionala table
and read from a temporary non-transactional table.
9. The mixed statement is about to update a temporary non-transactional
table and update a non-transactional table.
10. The mixed statement is about to update a temporary non-transactional
table and read from a non-transactional table.
11. A statement is about to update a non-transactional table and the
option variables.binlog_direct_non_trans_update is OFF.
The reason for this is that locks acquired may not protected a concurrent
transaction of interfering in the current execution and by consequence in
the result. So the patch reduced the number of spurious unsafe warnings.
Besides we fixed a regression caused by BUG#51894, which makes temporary
tables to go into the trx-cache if there is an on-going transaction. In
MIXED mode, the patch for BUG#51894 ignores that the trx-cache may have
updates to temporary non-transactional tables that must be written to the
binary log while rolling back the transaction.
So we fix this problem by writing the content of the trx-cache to the
binary log while rolling back a transaction if a non-transactional
temporary table was updated and the binary logging format is MIXED.
Conflicts:
Text conflict in mysql-test/r/archive.result
Contents conflict in mysql-test/r/innodb_bug38231.result
Text conflict in mysql-test/r/mdl_sync.result
Text conflict in mysql-test/suite/binlog/t/disabled.def
Text conflict in mysql-test/suite/rpl_ndb/r/rpl_ndb_binlog_format_errors.result
Text conflict in mysql-test/t/archive.test
Contents conflict in mysql-test/t/innodb_bug38231.test
Text conflict in mysql-test/t/mdl_sync.test
Text conflict in sql/sp_head.cc
Text conflict in sql/sql_show.cc
Text conflict in sql/table.cc
Text conflict in sql/table.h
This patch fixes two problems described as follows:
1 - If there is an on-going transaction and a temporary table is created or
dropped, any failed statement that follows the "create" or "drop commands"
triggers a rollback and by consequence the slave will go out sync because
the binary log will have a wrong sequence of events.
To fix the problem, we changed the expression that evaluates when the
cache should be flushed after either the rollback of a statment or
transaction.
2 - When a "CREATE TEMPORARY TABLE SELECT * FROM" was executed the
OPTION_KEEP_LOG was not set into the thd->options. For that reason, if
the transaction had updated only transactional engines and was rolled
back at the end (.e.g due to a deadlock) the changes were not written
to the binary log, including the creation of the temporary table.
To fix the problem, we have set the OPTION_KEEP_LOG into the thd->options
when a "CREATE TEMPORARY TABLE SELECT * FROM" is executed.
sql/log.cc:
Reorganized the code based on the following functions:
- bool ending_trans(const THD* thd, const bool all);
- bool trans_has_updated_non_trans_table(const THD* thd);
- bool trans_has_no_stmt_committed(const THD* thd, const bool all);
- bool stmt_has_updated_non_trans_table(const THD* thd);
sql/log.h:
Added functions to organize the code in log.cc.
sql/log_event.cc:
Removed the OPTION_KEEP_LOG since it must be used only when
creating and dropping temporary tables.
sql/log_event_old.cc:
Removed the OPTION_KEEP_LOG since it must be used only when
creating and dropping temporary tables.
sql/sql_parse.cc:
When a "CREATE TEMPORARY TABLE SELECT * FROM" was executed the
OPTION_KEEP_LOG was not set into the thd->options.
To fix the problem, we have set the OPTION_KEEP_LOG into the
thd->options when a "CREATE TEMPORARY TABLE SELECT * FROM"
is executed.
MYSQL_BIN_LOG m_table_map_version member and it's associated
functions were not used in the logic of binlogging and replication,
this patch removed all related code.
sql/log.cc:
removed unused m_table_map_version variable and functions
sql/log.h:
removed unused m_table_map_version variable and functions
sql/log_event.h:
Removed unused LOG_EVENT_UPDATE_TABLE_MAP_VERSION_F flag
sql/sql_class.cc:
Removed unused LOG_EVENT_UPDATE_TABLE_MAP_VERSION_F flag
sql/sql_load.cc:
Removed unused LOG_EVENT_UPDATE_TABLE_MAP_VERSION_F flag
sql/table.cc:
removed unused table_map_version variable
sql/table.h:
removed unused table_map_version variable
transaction
BUG#52616 Temp table prevents switch binlog format from STATEMENT to ROW
Before the WL#2687 and BUG#46364, every non-transactional change that happened
after a transactional change was written to trx-cache and flushed upon
committing the transaction. WL#2687 and BUG#46364 changed this behavior and
non-transactional changes are now written to the binary log upon committing
the statement.
A binary log event is identified as transactional or non-transactional through
a flag in the Log_event which is set taking into account the underlie storage
engine on what it is stems from. In the current bug, this flag was not being
set properly when the DROP TEMPORARY TABLE was executed.
However, while fixing this bug we figured out that changes to temporary tables
should be always written to the trx-cache if there is an on-going transaction.
Otherwise, binlog events in the reversed order would be produced.
Regarding concurrency, keeping changes to temporary tables in the trx-cache is
also safe as temporary tables are only visible to the owner connection.
In this patch, we classify the following statements as unsafe:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
On the other hand, the following statements are classified as safe:
1 - INSERT INTO t_innodb SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_innodb
The patch also guarantees that transactions that have a DROP TEMPORARY are
always written to the binary log regardless of the mode and the outcome:
commit or rollback. In particular, the DROP TEMPORARY is extended with the
IF EXISTS clause when the current statement logging format is set to row.
Finally, the patch allows to switch from STATEMENT to MIXED/ROW when there
are temporary tables but the contrary is not possible.
mysql-test/extra/rpl_tests/rpl_binlog_max_cache_size.test:
Updated the test case because
CREATE TEMPORARY TABLE t_innodb_temp SELECT * FROM t_myisam is not unsafe.
mysql-test/extra/rpl_tests/rpl_implicit_commit_binlog.test:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/extra/rpl_tests/rpl_innodb.test:
Removed comments from the test case that became false after the patch.
mysql-test/extra/rpl_tests/rpl_loaddata.test:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
mysql-test/include/ctype_utf8_table.inc:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
mysql-test/r/ctype_cp932_binlog_stm.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/binlog/r/binlog_database.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/binlog/r/binlog_innodb_row.result:
Updated the result file.
mysql-test/suite/binlog/r/binlog_multi_engine.result:
Updated the unsafe message.
mysql-test/suite/binlog/r/binlog_row_binlog.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/binlog/r/binlog_row_drop_tmp_tbl.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/binlog/r/binlog_stm_binlog.result:
Updated the result file.
mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/binlog/t/binlog_tmp_table.test:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
mysql-test/suite/ndb/r/ndb_binlog_format.result:
Updated the unsafe message.
mysql-test/suite/rpl/r/rpl_concurrency_error.result:
Updated the unsafe message.
mysql-test/suite/rpl/r/rpl_mixed_binlog_max_cache_size.result:
Updated the result file because
CREATE TEMPORARY TABLE t_innodb_temp SELECT * FROM t_myisam is not unsafe.
mysql-test/suite/rpl/r/rpl_mixed_implicit_commit_binlog.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_mixed_mixing_engines.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_mixed_row_innodb.result:
Added some comments to ease the understanding of the result file.
mysql-test/suite/rpl/r/rpl_non_direct_mixed_mixing_engines.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_non_direct_row_mixing_engines.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_non_direct_stm_mixing_engines.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_row_drop.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_row_implicit_commit_binlog.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_row_mixing_engines.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_stm_binlog_max_cache_size.result:
Updated the result file because
CREATE TEMPORARY TABLE t_innodb_temp SELECT * FROM t_myisam is not unsafe.
mysql-test/suite/rpl/r/rpl_stm_implicit_commit_binlog.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_stm_innodb.result:
Added some comments to ease the understanding of the result file.
mysql-test/suite/rpl/r/rpl_stm_mixing_engines.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_stm_stop_middle_group.result:
Updated the unsafe message.
mysql-test/suite/rpl/r/rpl_temp_temporary.result:
Added a test case.
mysql-test/suite/rpl/t/rpl000013.test:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
mysql-test/suite/rpl/t/rpl_misc_functions.test:
Suppressed warning messages.
mysql-test/suite/rpl/t/rpl_temp_table.test:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
mysql-test/suite/rpl/t/rpl_temp_temporary.test:
Added a test case.
mysql-test/suite/rpl/t/rpl_temporary.test:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
mysql-test/suite/rpl_ndb/r/rpl_ndb_row_implicit_commit_binlog.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl_ndb/r/rpl_truncate_7ndb.result:
Updated the test case to remove references to positions
in the binary log.
mysql-test/suite/rpl_ndb/t/rpl_truncate_7ndb.test:
Updated the test case to remove references to positions
in the binary log.
mysql-test/t/create_select_tmp.test:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
mysql-test/t/ctype_cp932_binlog_stm.test:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
mysql-test/t/mysqlbinlog.test:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
sql/log.cc:
Improved the code by creating several functions to hide decision
on type of engine changed, commit/abort, etc:
. stmt_has_updated_non_trans_table
. trans_has_updated_non_trans_table
. ending_trans
Updated the binlog_rollback function and the use of the
OPTION_KEEP_LOG which indincates when a temporary table was
either created or dropped and as such the command must be
logged if not in MIXED mode and even while rolling back the
transaction.
sql/log.h:
Improved the code by creating several functions to hide decision
on type of engine changed, commit/abort, etc.
sql/log_event.cc:
Removed the setting of the OPTION_KEEP_LOG as it is related to CREATE
TEMPORARY and DROP TEMPORARY and not to the type of engine (i.e.
transactional or non-transactional).
sql/log_event_old.cc:
Removed the setting of the OPTION_KEEP_LOG as it is related to CREATE
TEMPORARY and DROP TEMPORARY and not to the type of engine (i.e.
transactional or non-transactional).
sql/share/errmsg-utf8.txt:
Updated the unsafe message.
sql/sql_class.cc:
Classifies the following statements as unsafe:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
On the other hand, the following statements are classified as safe:
1 - INSERT INTO t_innodb SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_innodb
sql/sql_class.h:
It allows to switch from STATEMENT to MIXED/ROW when there are temporary
tables but the contrary is not possible.
sql/sql_table.cc:
Fixed the case that a DROP/DROP TEMPORARY that affects a temporary table in MIXED
mode is written as a DROP TEMPORARY TABLE IF EXISTS because the table may not exist in the slave and due to the IF EXISTS token an error will never happen
while processing the statement in the slave.
Removed a function that was not being used.
This patch:
- Moves all definitions from the mysql_priv.h file into
header files for the component where the variable is
defined
- Creates header files if the component lacks one
- Eliminates all include directives from mysql_priv.h
- Eliminates all circular include cycles
- Rename time.cc to sql_time.cc
- Rename mysql_priv.h to sql_priv.h
Conflicts:
- mysql-test/r/mysqld--help-win.result
- sql/sys_vars.cc
Original revsion (in next-mr-bugfixing):
------------------------------------------------------------
revno: 2971 [merge]
revision-id: alfranio.correia@sun.com-20100121210527-rbuheu5rnsmcakh1
committer: Alfranio Correia <alfranio.correia@sun.com>
branch nick: mysql-next-mr-bugfixing
timestamp: Thu 2010-01-21 21:05:27 +0000
message:
BUG#46364 MyISAM transbuffer problems (NTM problem)
It is well-known that due to concurrency issues, a slave can become
inconsistent when a transaction contains updates to both transaction and
non-transactional tables.
In a nutshell, the current code-base tries to preserve causality among the
statements by writing non-transactional statements to the txn-cache which
is flushed upon commit. However, modifications done to non-transactional
tables on behalf of a transaction become immediately visible to other
connections but may not immediately get into the binary log and therefore
consistency may be broken.
In general, it is impossible to automatically detect causality/dependency
among statements by just analyzing the statements sent to the server. This
happen because dependency may be hidden in the application code and it is
necessary to know a priori all the statements processed in the context of
a transaction such as in a procedure. Moreover, even for the few cases that
we could automatically address in the server, the computation effort
required could make the approach infeasible.
So, in this patch we introduce the option
- "--binlog-direct-non-transactional-updates" that can be used to bypass
the current behavior in order to write directly to binary log statements
that change non-transactional tables.
Besides, it is used to enable the WL#2687 which is disabled by default.
------------------------------------------------------------
revno: 2970.1.1
revision-id: alfranio.correia@sun.com-20100121131034-183r4qdyld7an5a0
parent: alik@sun.com-20100121083914-r9rz2myto3tkdya0
committer: Alfranio Correia <alfranio.correia@sun.com>
branch nick: mysql-next-mr-bugfixing
timestamp: Thu 2010-01-21 13:10:34 +0000
message:
BUG#46364 MyISAM transbuffer problems (NTM problem)
It is well-known that due to concurrency issues, a slave can become
inconsistent when a transaction contains updates to both transaction and
non-transactional tables.
In a nutshell, the current code-base tries to preserve causality among the
statements by writing non-transactional statements to the txn-cache which
is flushed upon commit. However, modifications done to non-transactional
tables on behalf of a transaction become immediately visible to other
connections but may not immediately get into the binary log and therefore
consistency may be broken.
In general, it is impossible to automatically detect causality/dependency
among statements by just analyzing the statements sent to the server. This
happen because dependency may be hidden in the application code and it is
necessary to know a priori all the statements processed in the context of
a transaction such as in a procedure. Moreover, even for the few cases that
we could automatically address in the server, the computation effort
required could make the approach infeasible.
So, in this patch we introduce the option
- "--binlog-direct-non-transactional-updates" that can be used to bypass
the current behavior in order to write directly to binary log statements
that change non-transactional tables.
Besides, it is used to enable the WL#2687 which is disabled by default.
Conflicts:
Text conflict in .bzr-mysql/default.conf
Text conflict in mysql-test/extra/rpl_tests/rpl_loaddata.test
Text conflict in mysql-test/r/mysqlbinlog2.result
Text conflict in mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result
Text conflict in mysql-test/suite/binlog/r/binlog_unsafe.result
Text conflict in mysql-test/suite/rpl/r/rpl_insert_id.result
Text conflict in mysql-test/suite/rpl/r/rpl_loaddata.result
Text conflict in mysql-test/suite/rpl/r/rpl_stm_auto_increment_bug33029.result
Text conflict in mysql-test/suite/rpl/r/rpl_udf.result
Text conflict in mysql-test/suite/rpl/t/rpl_slow_query_log.test
Text conflict in sql/field.h
Text conflict in sql/log.cc
Text conflict in sql/log_event.cc
Text conflict in sql/log_event_old.cc
Text conflict in sql/mysql_priv.h
Text conflict in sql/share/errmsg.txt
Text conflict in sql/sp.cc
Text conflict in sql/sql_acl.cc
Text conflict in sql/sql_base.cc
Text conflict in sql/sql_class.h
Text conflict in sql/sql_db.cc
Text conflict in sql/sql_delete.cc
Text conflict in sql/sql_insert.cc
Text conflict in sql/sql_lex.cc
Text conflict in sql/sql_lex.h
Text conflict in sql/sql_load.cc
Text conflict in sql/sql_table.cc
Text conflict in sql/sql_update.cc
Text conflict in sql/sql_view.cc
Conflict adding files to storage/innobase. Created directory.
Conflict because storage/innobase is not versioned, but has versioned children. Versioned directory.
Conflict adding file storage/innobase. Moved existing file to storage/innobase.moved.
Conflict adding files to storage/innobase/handler. Created directory.
Conflict because storage/innobase/handler is not versioned, but has versioned children. Versioned directory.
Contents conflict in storage/innobase/handler/ha_innodb.cc
Bug#16565 mysqld --help --verbose does not order variablesBug#20413 sql_slave_skip_counter is not shown in show variables
Bug#20415 Output of mysqld --help --verbose is incomplete
Bug#25430 variable not found in SELECT @@global.ft_max_word_len;
Bug#32902 plugin variables don't know their names
Bug#34599 MySQLD Option and Variable Reference need to be consistent in formatting!
Bug#34829 No default value for variable and setting default does not raise error
Bug#34834 ? Is accepted as a valid sql mode
Bug#34878 Few variables have default value according to documentation but error occurs
Bug#34883 ft_boolean_syntax cant be assigned from user variable to global var.
Bug#37187 `INFORMATION_SCHEMA`.`GLOBAL_VARIABLES`: inconsistent status
Bug#40988 log_output_basic.test succeeded though syntactically false.
Bug#41010 enum-style command-line options are not honoured (maria.maria-recover fails)
Bug#42103 Setting key_buffer_size to a negative value may lead to very large allocations
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
Bug#44797 plugins w/o command-line options have no disabling option in --help
Bug#46314 string system variables don't support expressions
Bug#46470 sys_vars.max_binlog_cache_size_basic_32 is broken
Bug#46586 When using the plugin interface the type "set" for options caused a crash.
Bug#47212 Crash in DBUG_PRINT in mysqltest.cc when trying to print octal number
Bug#48758 mysqltest crashes on sys_vars.collation_server_basic in gcov builds
Bug#49417 some complaints about mysqld --help --verbose output
Bug#49540 DEFAULT value of binlog_format isn't the default value
Bug#49640 ambiguous option '--skip-skip-myisam' (double skip prefix)
Bug#49644 init_connect and \0
Bug#49645 init_slave and multi-byte characters
Bug#49646 mysql --show-warnings crashes when server dies
CMakeLists.txt:
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
client/mysql.cc:
don't crash with --show-warnings when mysqld dies
config/ac-macros/plugins.m4:
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
include/my_getopt.h:
comments
include/my_pthread.h:
fix double #define
mysql-test/mysql-test-run.pl:
run sys_vars suite by default
properly recognize envirinment variables (e.g. MTR_MAX_SAVE_CORE) set to 0
escape gdb command line arguments
mysql-test/suite/sys_vars/r/rpl_init_slave_func.result:
init_slave+utf8 bug
mysql-test/suite/sys_vars/t/rpl_init_slave_func.test:
init_slave+utf8 bug
mysys/my_getopt.c:
Bug#34599 MySQLD Option and Variable Reference need to be consistent in formatting!
Bug#46586 When using the plugin interface the type "set" for options caused a crash.
Bug#49640 ambiguous option '--skip-skip-myisam' (double skip prefix)
mysys/typelib.c:
support for flagset
sql/ha_ndbcluster.cc:
backport from telco tree
sql/item_func.cc:
Bug#49644 init_connect and \0
Bug#49645 init_slave and multi-byte characters
sql/sql_builtin.cc.in:
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
sql/sql_plugin.cc:
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
Bug#32902 plugin variables don't know their names
Bug#44797 plugins w/o command-line options have no disabling option in --help
sql/sys_vars.cc:
all server variables are defined here
storage/myisam/ft_parser.c:
remove unnecessary updates of param->quot
storage/myisam/ha_myisam.cc:
myisam_* variables belong here
strings/my_vsnprintf.c:
%o and %llx
unittest/mysys/my_vsnprintf-t.c:
%o and %llx tests
vio/viosocket.c:
bugfix: fix @@wait_timeout to work with socket timeouts (vs. alarm thread)
This patch fixes three bugs as follows. First, aborting the server while purging
binary logs might generate orphan files due to how the purge operation was
implemented:
(purge routine - sql/log.cc - MYSQL_BIN_LOG::purge_logs)
1 - register the files to be removed in a temporary buffer.
2 - update the log-bin.index.
3 - flush the log-bin.index.
4 - erase the files whose names where register in the temporary buffer
in step 1.
Thus a failure while executing step 4 would generate an orphan file. Second,
a similar issue might happen while creating a new binary as follows:
(create routine - sql/log.cc - MYSQL_BIN_LOG::open)
1 - open the new log-bin.
2 - update the log-bin.index.
Thus a failure while executing step 1 would generate an orphan file.
To fix these issues, we record the files to be purged or created before really
removing or adding them. So if a failure happens such records can be used to
automatically remove dangling files. The new steps might be outlined as follows:
(purge routine - sql/log.cc - MYSQL_BIN_LOG::purge_logs)
1 - register the files to be removed in the log-bin.~rec~ placed in
the data directory.
2 - update the log-bin.index.
3 - flush the log-bin.index.
4 - delete the log-bin.~rec~.
(create routine - sql/log.cc - MYSQL_BIN_LOG::open)
1 - register the file to be created in the log-bin.~rec~
placed in the data directory.
2 - open the new log-bin.
3 - update the log-bin.index.
4 - delete the log-bin.~rec~.
(recovery routine - sql/log.cc - MYSQL_BIN_LOG::open_index_file)
1 - open the log-bin.index.
2 - open the log-bin.~rec~.
3 - for each file in log-bin.~rec~.
3.1 Check if the file is in the log-bin.index and if so ignore it.
3.2 Otherwise, delete it.
The third issue can be described as follows. The purge operation was allowing
to remove a file in use thus leading to the loss of data and possible
inconsistencies between the master and slave. Roughly, the routine was only
taking into account the dump threads and so if a slave was not connect the
file might be delete even though it was in use.
Support for flushing individual logs, so that the user can
selectively flush a subset of the server logs.
Flush of individual logs is done according to the
following syntax:
FLUSH <log_category> LOGS;
The syntax is extended so that the user is able to flush a
subset of logs:
FLUSH [log_category LOGS,];
where log_category is one of:
SLOW
ERROR
BINARY
ENGINE
GENERAL
RELAY.
mysql-test/suite/rpl/r/rpl_flush_logs.result:
Test result for WL#5142.
mysql-test/suite/rpl/t/rpl_flush_logs.test:
Added the test file to verify if the 'flush individual log'
statement works fine.
sql/log.cc:
Added the two functions to flush slow and general log.
sql/sql_parse.cc:
Added code to flush specified logs against the option.
sql/sql_yacc.yy:
Added code to parse the 'flush * log' statement syntax and
set its option to Lex->type.
beyond unsigned long.
BUG#44779: binlog.binlog_max_extension may be causing failure on
next test in PB
NOTE1: this is the backport to next-mr.
NOTE2: already includes patch for BUG#44779.
Binlog file extensions would turn into negative numbers once the
variable used to hold the value reached maximum for signed
long. Consequently, incrementing value to the next (negative) number
would lead to .000000 extension, causing the server to fail.
This patch addresses this issue by not allowing negative extensions
and by returning an error on find_uniq_filename, when the limit is
reached. Additionally, warnings are printed to the error log when the
limit is approaching. FLUSH LOGS will also report warnings to the
user, if the extension number has reached the limit. The limit has been
set to 0x7FFFFFFF as the maximum.
mysql-test/suite/binlog/t/binlog_max_extension.test:
Test case added that checks the maximum available number for
binlog extensions.
sql/log.cc:
Changes to find_uniq_filename and test_if_number.
sql/log.h:
Added macros with values for MAX_LOG_UNIQUE_FN_EXT and
LOG_WARN_UNIQUE_FN_EXT_LEFT, as suggested in review.
The mentioned on the bug report set of bugs fixes have not be pushed to the main trees.
Fixed with extracting commits done to 6.0-rpl tree and applying them to the main 5.1.
Notes.
1. part of changes - the mtr's specific - were packported to the main 5.0 tree for mtr v1
as http://lists.mysql.com/commits/46562
However, there is no that fix anymore in the mtr v2. (This fact was mailed to mtr maintaining
people).
2. Bug@36929 crash in kill_zombie_dump_threads-> THD::awake() with replication tests
is not backported because the base code of the patch is libevent and that was removed
from the main trees due to its instability.
client/mysqlbinlog.cc:
fixes for BUG#35546
mysql-test/suite/rpl/r/rpl_bug41902.result:
the new tests result file is added.
mysql-test/suite/rpl/t/rpl_bug41902-slave.opt:
conf file for bug41902 testing is added.
mysql-test/suite/rpl/t/rpl_bug41902.test:
regression tests for Bug #41902 is added.
sql/log.cc:
collection of changes due to Bug #48463.
sql/log.h:
collection of changes due to Bug #48463.
sql/rpl_rli.h:
collection of changes due to Bug #48463.
sql/slave.cc:
collection of changes due to Bug #48463.
sql/sql_repl.cc:
collection of changes due to Bug #48463.
Non-transactional updates that take place inside a transaction present problems
for logging because they are visible to other clients before the transaction
is committed, and they are not rolled back even if the transaction is rolled
back. It is not always possible to log correctly in statement format when both
transactional and non-transactional tables are used in the same transaction.
In the current patch, we ensure that such scenario is completely safe under the
ROW and MIXED modes.
The original code comes, as far as I know, from Google (Mark Callaghan's team) with additional work from Percona, Ourdelta and Weldon Whipple.
This code provides the same functionallity, but with a lot of changes to make it faster and better fit the MariaDB infrastucture.
Added new status variables:
- Com_show_client_statistics, Com_show_index_statistics, Com_show_table_statistics, Com_show_user_statistics
- Access_denied_errors, Busy_time (clock time), Binlog_bytes_written, Cpu_time, Empty_queries, Rows_sent, Rows_read
Added new variable / startup option 'userstat' to control if user statistics should be enabled or not
Added my_getcputime(); Returns cpu time used by this thread.
New FLUSH commands:
- FLUSH SLOW QUERY LOG
- FLUSH TABLE_STATISTICS
- FLUSH INDEX_STATISTICS
- FLUSH USER_STATISTICS
- FLUSH CLIENT_STATISTICS
New SHOW commands:
- SHOW CLIENT_STATISTICS
- SHOW USER_STATISTICS
- SHOW TABLE_STATISTICS
- SHOW INDEX_STATISTICS
New Information schemas:
- CLIENT_STATISTICS
- USER_STATISTICS
- INDEX_STATISTICS
- TABLE_STATISTICS
Added support for all new flush commands to mysqladmin
Added handler::ha_... wrappers for all handler read calls to do statistics counting
- Changed all code to use new ha_... calls
- Count number of read rows, changed rows and rows read trough an index
Added counting of number of bytes sent to binary log (status variable Binlog_bytes_written)
Added counting of access denied errors (status variable Access_denied_erors)
Bugs fixed:
- Fixed bug in add_to_status() and add_diff_to_status() where longlong variables where threated as long
- CLOCK_GETTIME was not propely working on Linuxm
client/mysqladmin.cc:
Added support for all new flush commmands and some common combinations:
flush-slow-log
flush-table-statistics
flush-index-statistics
flush-user-statistics
flush-client-statistics
flush-all-status
flush-all-statistics
configure.in:
Added checking if clock_gettime needs the librt.
(Fixes Bug #37639 clock_gettime is never used/enabled in Linux/Unix)
include/my_sys.h:
Added my_getcputime()
include/mysql_com.h:
Added LIST_PROCESS_HOST_LEN & new REFRESH target defines
mysql-test/r/information_schema.result:
New information schema tables added
mysql-test/r/information_schema_all_engines.result:
New information schema tables added
mysql-test/r/information_schema_db.result:
New information schema tables added
mysql-test/r/log_slow.result:
Added testing that flosh slow query logs is accepted
mysql-test/r/status_user.result:
Basic testing of user, client, table and index statistics
mysql-test/t/log_slow.test:
Added testing that flosh slow query logs is accepted
mysql-test/t/status_user-master.opt:
Ensure that we get a fresh restart before running status_user.test
mysql-test/t/status_user.test:
Basic testing of user, client, table and index statistics
mysys/my_getsystime.c:
Added my_getcputime()
Returns cpu time used by this thread.
sql/authors.h:
Updated authors to have core and original MySQL developers first.
sql/event_data_objects.cc:
Updated call to mysql_reset_thd_for_next_command()
sql/event_db_repository.cc:
Changed to use new ha_... calls
sql/filesort.cc:
Changed to use new ha_... calls
sql/ha_partition.cc:
Changed to use new ha_... calls
Fixed comment syntax
sql/handler.cc:
Changed to use new ha_... calls
Reset table statistics
Added code to update global table and index status
Added counting of rows changed
sql/handler.h:
Added table and index statistics variables
Added function reset_statistics()
Added handler::ha_... wrappers for all handler read calls to do statistics counting
Protected all normal read calls to ensure we use the new calls in the server.
Made ha_partition a friend class so that partition code can call the old read functions
sql/item_subselect.cc:
Changed to use new ha_... calls
sql/lex.h:
Added keywords for new information schema tables and flush commands
sql/log.cc:
Added flush_slow_log()
Added counting of number of bytes sent to binary log
Removed not needed test of thd (It's used before, so it's safe to use)
Added THD object to MYSQL_BIN_LOG::write_cache() to simplify statistics counting
sql/log.h:
Added new parameter to write_cache()
Added flush_slow_log() functions.
sql/log_event.cc:
Updated call to mysql_reset_thd_for_next_command()
Changed to use new ha_... calls
sql/log_event_old.cc:
Updated call to mysql_reset_thd_for_next_command()
Changed to use new ha_... calls
sql/mysql_priv.h:
Updated call to mysql_reset_thd_for_next_command()
Added new statistics functions and variables needed by these.
sql/mysqld.cc:
Added new statistics variables and structures to handle these
Added new status variables:
- Com_show_client_statistics, Com_show_index_statistics, Com_show_table_statistics, Com_show_user_statistics
- Access_denied_errors, Busy_time (clock time), Binlog_bytes_written, Cpu_time, Empty_queries, Rows_set, Rows_read
Added new option 'userstat' to control if user statistics should be enabled or not
sql/opt_range.cc:
Changed to use new ha_... calls
sql/opt_range.h:
Changed to use new ha_... calls
sql/opt_sum.cc:
Changed to use new ha_... calls
sql/records.cc:
Changed to use new ha_... calls
sql/set_var.cc:
Added variable 'userstat'
sql/sp.cc:
Changed to use new ha_... calls
sql/sql_acl.cc:
Changed to use new ha_... calls
Added counting of access_denied_errors
sql/sql_base.cc:
Added call to statistics functions
sql/sql_class.cc:
Added usage of org_status_var, to store status variables at start of command
Added functions THD::update_stats(), THD::update_all_stats()
Fixed bug in add_to_status() and add_diff_to_status() where longlong variables where threated as long
sql/sql_class.h:
Added new status variables to status_var
Moved variables that was not ulong in status_var last.
Added variables to THD for storing temporary values during statistics counting
sql/sql_connect.cc:
Variables and functions to calculate user and client statistics
Added counting of access_denied_errors and lost_connections
sql/sql_cursor.cc:
Changed to use new ha_... calls
sql/sql_handler.cc:
Changed to use new ha_... calls
sql/sql_help.cc:
Changed to use new ha_... calls
sql/sql_insert.cc:
Changed to use new ha_... calls
sql/sql_lex.h:
Added SQLCOM_SHOW_USER_STATS, SQLCOM_SHOW_TABLE_STATS, SQLCOM_SHOW_INDEX_STATS, SQLCOM_SHOW_CLIENT_STATS
sql/sql_parse.cc:
Added handling of:
- SHOW CLIENT_STATISTICS
- SHOW USER_STATISTICS
- SHOW TABLE_STATISTICS
- SHOW INDEX_STATISTICS
Added handling of new FLUSH commands:
- FLUSH SLOW QUERY LOGS
- FLUSH TABLE_STATISTICS
- FLUSH INDEX_STATISTICS
- FLUSH USER_STATISTICS
- FLUSH CLIENT_STATISTICS
Added THD parameter to mysql_reset_thd_for_next_command()
Added initialization and calls to user statistics functions
Added increment of statistics variables empty_queries, rows_sent and access_denied_errors.
Added counting of cpu time per query
sql/sql_plugin.cc:
Changed to use new ha_... calls
sql/sql_prepare.cc:
Updated call to mysql_reset_thd_for_next_command()
sql/sql_select.cc:
Changed to use new ha_... calls
Indentation changes
sql/sql_servers.cc:
Changed to use new ha_... calls
sql/sql_show.cc:
Added counting of access denied errors
Added function for new information schema tables:
- CLIENT_STATISTICS
- USER_STATISTICS
- INDEX_STATISTICS
- TABLE_STATISTICS
Changed to use new ha_... calls
sql/sql_table.cc:
Changed to use new ha_... calls
sql/sql_udf.cc:
Changed to use new ha_... calls
sql/sql_update.cc:
Changed to use new ha_... calls
sql/sql_yacc.yy:
Add new show and flush commands
sql/structs.h:
Add name_length to KEY to avoid some strlen
Added cache_name to KEY for fast storage of keyvalue in cache
Added structs USER_STATS, TABLE_STATS, INDEX_STATS
Added function prototypes for statistics functions
sql/table.cc:
Store db+table+index name into keyinfo->cache_name
sql/table.h:
Added new information schema tables
sql/tztime.cc:
Changed to use new ha_... calls
BUG#31665 sync_binlog should cause relay logs to be synchronized
NOTE: Backporting the patch to next-mr.
Add sync_relay_log option to server, this option works for relay log
the same as option sync_binlog for binlog. This option also synchronize
master info to disk when set to non-zero value.
Original patches from Sinisa and Mark, with some modifications
beyond unsigned long.
BUG#44779: binlog.binlog_max_extension may be causing failure on
next test in PB
NOTE1: this is the backport to next-mr.
NOTE2: already includes patch for BUG#44779.
Binlog file extensions would turn into negative numbers once the
variable used to hold the value reached maximum for signed
long. Consequently, incrementing value to the next (negative) number
would lead to .000000 extension, causing the server to fail.
This patch addresses this issue by not allowing negative extensions
and by returning an error on find_uniq_filename, when the limit is
reached. Additionally, warnings are printed to the error log when the
limit is approaching. FLUSH LOGS will also report warnings to the
user, if the extension number has reached the limit. The limit has been
set to 0x7FFFFFFF as the maximum.
mysql-test/suite/binlog/t/binlog_max_extension.test:
Test case added that checks the maximum available number for
binlog extensions.
sql/log.cc:
Changes to find_uniq_filename and test_if_number.
sql/log.h:
Added macros with values for MAX_LOG_UNIQUE_FN_EXT and
LOG_WARN_UNIQUE_FN_EXT_LEFT, as suggested in review.
Large transactions and statements may corrupt the binary log if the size of the
cache, which is set by the max_binlog_cache_size, is not enough to store the
the changes.
In a nutshell, to fix the bug, we save the position of the next character in the
cache before starting processing a statement. If there is a problem, we simply
restore the position thus removing any effect of the statement from the cache.
Unfortunately, to avoid corrupting the binary log, we may end up loosing changes
on non-transactional tables if they do not fit in the cache. In such cases, we
store an Incident_log_event in order to stop the slave and alert users that some
changes were not logged.
Precisely, for every non-transactional changes that do not fit into the cache,
we do the following:
a) the statement is *not* logged
b) an incident event is logged after committing/rolling back the transaction,
if any. Note that if a failure happens before writing the incident event to
the binary log, the slave will not stop and the master will not have reported
any error.
c) its respective statement gives an error
For transactional changes that do not fit into the cache, we do the following:
a) the statement is *not* logged
b) its respective statement gives an error
To work properly, this patch requires two additional things. Firstly, callers to
MYSQL_BIN_LOG::write and THD::binlog_query must handle any error returned and
take the appropriate actions such as undoing the effects of a statement. We
already changed some calls in the sql_insert.cc, sql_update.cc and sql_insert.cc
modules but the remaining calls spread all over the code should be handled in
BUG#37148. Secondly, statements must be either classified as DDL or DML because
DDLs that do not get into the cache must generate an incident event since they
cannot be rolled back.
Make the caller of Query_log_event, Execute_load_log_event
constructors and THD::binlog_query to provide the error code
instead of having the constructors to figure out the error code.
sql/log_event.cc:
Changed constructors of Query_log_event and Execute_load_log_event to accept the error code argument instead of figuring it out by itself
sql/log_event.h:
Changed constructors of Query_log_event and Execute_load_log_event to accept the error code argument
conflicts:
Text conflict in client/mysqltest.cc
Text conflict in mysql-test/include/wait_until_connected_again.inc
Text conflict in mysql-test/lib/mtr_report.pm
Text conflict in mysql-test/mysql-test-run.pl
Text conflict in mysql-test/r/events_bugs.result
Text conflict in mysql-test/r/log_state.result
Text conflict in mysql-test/r/myisam_data_pointer_size_func.result
Text conflict in mysql-test/r/mysqlcheck.result
Text conflict in mysql-test/r/query_cache.result
Text conflict in mysql-test/r/status.result
Text conflict in mysql-test/suite/binlog/r/binlog_index.result
Text conflict in mysql-test/suite/binlog/r/binlog_innodb.result
Text conflict in mysql-test/suite/rpl/r/rpl_packet.result
Text conflict in mysql-test/suite/rpl/t/rpl_packet.test
Text conflict in mysql-test/t/disabled.def
Text conflict in mysql-test/t/events_bugs.test
Text conflict in mysql-test/t/log_state.test
Text conflict in mysql-test/t/myisam_data_pointer_size_func.test
Text conflict in mysql-test/t/mysqlcheck.test
Text conflict in mysql-test/t/query_cache.test
Text conflict in mysql-test/t/rpl_init_slave_func.test
Text conflict in mysql-test/t/status.test
BUG#39325 Server crash inside MYSQL_LOG::purge_first_log halts replicaiton
The patch reverses the order of the purging and updating events for log and
relay-log.info/index files respectively.
This solves the problem of having holes caused by crashes happening between updating
info/index files and purging logs.
This patch also contains an aditional test case for testing the crashing before purge logs.
NOTE1: This is a combined patch for BUG#38826 and BUG#39325. This patch is based on
bugteam tree and takes into account reviewers suggestions.
NOTE2: Merge from 5.0-bugteam
Rotate event is automatically generated and written when rotating binary
log or relay log. Rotate events for relay logs are usually ignored by slave
SQL thread becuase they have the same server id as that of the slave.
However, if --replicate-same-server-id is enabled, rotate event
for relay log would be treated as if it's a rotate event from master, and
would be executed by slave to update the rli->group_master_log_name and
rli->group_master_log_pos to a wrong value and cause the MASTER_POS_WAIT
function to fail and return NULL.
This patch fixed this problem by setting a flag bit (LOG_EVENT_RELAY_LOG_F)
in the event to tell the SQL thread to ignore these Rotate events generated
for relay logs.
This patch also added another binlog event flag bit (LOG_EVENT_ARTIFICIAL_F)
to distinquish faked events, the method used before this was by checking if
log_pos was zero.
sql/log.h:
Add a member to MYSQL_BIN_LOG to distinguish binary log from relay log.
sql/log_event.cc:
Change artificial_event member to LOG_EVENT_ARTIFICIAL_F flag
If LOG_EVENT_RELAY_LOG_F is set in the event flags for a rotate event, ignore it when updating position
Refactored the code in Rotate_log_event::do_update_pos
sql/log_event.h:
Add LOG_EVENT_RELAY_LOG_F flag to Log_event flags
Add RELAY_LOG flag to Rotate_log_event flags
sql/sql_repl.cc:
Set LOG_EVENT_ARTIFICIAL_F for fake rotate events
