The function mysql_show_binlog_events has a local stack variable
'LOG_INFO linfo;', which is assigned to thd->current_linfo, however
this variable goes out of scope and is destroyed before clean
thd->current_linfo.
The problem is solved by moving 'LOG_INFO linfo;' to function scope.
BIN LOG HAS BEEN MOVED
When moving the binary/relay log files from one location to
another and restarting the server with a different log-bin or
relay-log paths, would cause the startup process to abort. The
root cause was that the server would not be able to find the log
files because it would consider old paths for entries in the
index file instead of the new location. What's even worse, the
relative paths would not be considered relative to the path
provided in log-bin and relay-log, but to mysql_data_dir.
We fix the cases where the server contains relative paths. When
the server is reading from the index file, it checks whether the
entry contains relative paths. If it does, we replace it with the
absolute path set in log-bin/relay-log option. Absolute paths
remain unchanged and the index must be manually edited to
consider the new log-bin and/or relay-log path (this should be
documented). This is a fix for a GA version, that does not break
behavior (that much).
For development versions, we should go with Zhenxing's approach
that removes paths altogether from index files.
mysql-test/include/begin_include_file.inc:
Added parameter to keep the begin_include_file.inc silent. Useful when
including scripts that contain platform dependent parameters, for example:
--let $rpl_server_parameters=--log-bin=$tmpdir/slave-bin --relay-log=$tmpdir/slave-relay-bin
--let $keep_include_silent=1
source include/rpl_start_server.inc;
--let $keep_include_silent=0
We want the paths ($tmpdir/slave-bin and $tmpdir/slave-relay-bin) not to be in the
result file.
mysql-test/suite/rpl/t/rpl_binlog_index.test:
Test case.
sql/log.cc:
When finding the corresponding log entry in the index file, we first
normalize the paths before doing the comparison. This will make relative
paths to be turned into absolute paths (based on the opt_bin_logname or
opt_relay_logname) and then compared against also, expanded paths entered,
through CHANGE MASTER for instance.
sql/log.h:
Added normalize_binlog_name, which turns relative paths, into absolute paths
given the parameter: is_relay_log ? opt_relay_logname : opt_bin_logname .
sql/mysqld.cc:
Exposing opt_bin_logname.
sql/mysqld.h:
Exposing opt_bin_logname.
The bug case is similar to one fixed earlier bug_49536.
Deadlock involving LOCK_log appears to be possible because the purge running thread
is holding LOCK_log whereas there is no sense of doing that and which fact was
exploited by the earlier bug fixes.
Fixed with small reengineering of rotate_and_purge(), adding two new methods and
setting up a policy to execute those instead of the former
rotate_and_purge(RP_LOCK_LOG_IS_ALREADY_LOCKED).
The policy for using rotate(), purge() is that if the caller acquires LOCK_log itself,
it should call rotate(), release the mutex and run purge().
Side effect of this patch is refining error message of bug@11747416 to print
the whole path.
mysql-test/suite/rpl/r/rpl_cant_read_event_incident.result:
the file name printing is changed to a relative path instead of just the file name.
mysql-test/suite/rpl/r/rpl_log_pos.result:
the file name printing is changed to a relative path instead of just the file name.
mysql-test/suite/rpl/r/rpl_manual_change_index_file.result:
the file name printing is changed to a relative path instead of just the file name.
mysql-test/suite/rpl/r/rpl_packet.result:
the file name printing is changed to a relative path instead of just the file name.
mysql-test/suite/rpl/r/rpl_rotate_purge_deadlock.result:
new result file is added.
mysql-test/suite/rpl/t/rpl_cant_read_event_incident.test:
The test of that bug can't satisfy windows and unix backslash interpretation so windows
execution is chosen to bypass.
mysql-test/suite/rpl/t/rpl_rotate_purge_deadlock-master.opt:
new opt file is added.
mysql-test/suite/rpl/t/rpl_rotate_purge_deadlock.test:
regression test is added as well as verification of a
possible side effect of the fixes is tried.
sql/log.cc:
LOCK_log is never taken during execution of log purging routine.
The former MYSQL_BIN_LOG::rotate_and_purge is made to necessarily
acquiring and releasing LOCK_log.
If caller takes the mutex itself it has to use a new rotate(), purge()
methods combination and to never let purge() be run with LOCK_log grabbed.
split apart to allow
the caller to chose either it
Simulation of concurrently rotating/purging threads is added.
sql/log.h:
new rotate(), purge() methods are added to be used instead of
the former rotate_and_purge(RP_LOCK_LOG_IS_ALREADY_LOCKED).
rotate_and_purge() signature is changed. Caller should not call rotate_and_purge()
but rather {rotate(), purge()} if LOCK_log is acquired by it.
sql/rpl_injector.cc:
changes to reflect the new rotate_and_purge() signature.
sql/sql_class.h:
unnecessary constants are removed.
sql/sql_parse.cc:
changes to reflect the new rotate_and_purge() signature.
sql/sql_reload.cc:
changes to reflect the new rotate_and_purge() signature.
sql/sql_repl.cc:
followup for bug@11747416: the file name printing is changed to a relative
path instead of just the file name.
Before this fix, all the performance schema instrumentation for both the binary log
and the relay log would use the following instruments:
- wait/io/file/sql/binlog
- wait/io/file/sql/binlog_index
- wait/synch/mutex/sql/MYSQL_BIN_LOG::LOCK_index
- wait/synch/cond/sql/MYSQL_BIN_LOG::update_cond
This instrumentation is too general and can be more specific.
With this fix, the binlog instrumentation is identical,
and the relay log instrumentation is changed to:
- wait/io/file/sql/relaylog
- wait/io/file/sql/relaylog_index
- wait/synch/mutex/sql/MYSQL_RELAY_LOG::LOCK_index
- wait/synch/cond/sql/MYSQL_RELAY_LOG::update_cond
With this change, the performance instrumentation for the binary log and the relay log,
which share the same structure but have different uses, is more detailed.
This is especially important for hosts in the middle of a replication chain,
that are both masters (binlog) and slaves (relaylog).
Manual merge from mysql-5.1-bugteam into mysql-5.5-bugteam.
Conflicts
=========
Text conflict in sql/log.cc
Text conflict in sql/log.h
Text conflict in sql/slave.cc
Text conflict in sql/sql_parse.cc
Text conflict in sql/sql_priv.h
when generating new name.
If find_uniq_filename returns an error, then this error is not
being propagated upwards, and execution does not report error to
the user (although a entry in the error log is generated).
Additionally, some more errors were ignored in new_file_impl:
- when writing the rotate event
- when reopening the index and binary log file
This patch addresses this by propagating the error up in the
execution stack. Furthermore, when rotation of the binary log
fails, an incident event is written, because there may be a
chance that some changes for a given statement, were not properly
logged. For example, in SBR, LOAD DATA INFILE statement requires
more than one event to be logged, should rotation fail while
logging part of the LOAD DATA events, then the logged data would
become inconsistent with the data in the storage engine.
mysql-test/include/restart_mysqld.inc:
Refactored restart_mysqld so that it is not hardcoded for
mysqld.1, but rather for the current server.
mysql-test/suite/binlog/t/binlog_index.test:
The error on open of index and binary log on new_file_impl
is now caught. Thence the user will get an error message.
We need to accomodate this change in the test case for the
failing FLUSH LOGS.
mysql-test/suite/rpl/t/rpl_binlog_errors-master.opt:
Sets max_binlog_size to 4096.
mysql-test/suite/rpl/t/rpl_binlog_errors.test:
Added some test cases for asserting that the error is found
and reported.
sql/handler.cc:
Catching error now returned by unlog (in ha_commit_trans) and
returning it.
sql/log.cc:
Propagating errors from new_file_impl upwards. The errors that
new_file_impl catches now are:
- error on generate_new_name
- error on writing the rotate event
- error when opening the index or the binary log file.
sql/log.h:
Changing declaration of:
- rotate_and_purge
- new_file
- new_file_without_locking
- new_file_impl
- unlog
They now return int instead of void.
sql/mysql_priv.h:
Change signature of reload_acl_and_cache so that write_to_binlog
is an int instead of bool.
sql/mysqld.cc:
Redeclaring not_used var as int instead of bool.
sql/rpl_injector.cc:
Changes to catch the return from rotate_and_purge.
sql/slave.cc:
Changes to catch the return values for new_file and rotate_relay_log.
sql/slave.h:
Changes to rotate_relay_log declaration (now returns int
instead of void).
sql/sql_load.cc:
In SBR, some logging of LOAD DATA events goes through
IO_CACHE_CALLBACK invocation at mf_iocache.c:_my_b_get. The
IO_CACHE implementation is ignoring the return value for from
these callbacks (pre_read and post_read), so we need to find out
at the end of the execution if the error is set or not in THD.
sql/sql_parse.cc:
Catching the rotate_relay_log and rotate_and_purge return values.
Semantic change in reload_acl_and_cache so that we report errors
in binlog interactions through the write_to_binlog output parameter.
If there was any failure while rotating the binary log, we should
then report the error to the client when handling SQLCOMM_FLUSH.
Before this fix, file io for the binary log file was not accounted properly,
and showed no io at all.
This bug was due to the following issues:
1) file io for the binlog was instrumented:
- sometime as "wait/io/file/sql/binlog"
- sometime as "wait/io/file/sql/MYSQL_LOG"
leading to inconsistent event_names.
2) the binlog file itself was using an IO_CACHE,
but the IO_CACHE implementation in mysys/mf_iocache.c was
not instrumented to make performance schema calls to record file io.
3) The "wait/io/file/sql/MYSQL_LOG" instrumentation was used
for several log files, such as:
- the binary log
- the slow log
- the query log
which caused file io in these different log files to be accounted
against the same instrument.
The instrumentation needs to have a finer grain and report io
in different event_names, because each file really serves a different purpose.
With this fix:
- the IO_CACHE implementation is now instrumented
- the "wait/io/file/sql/MYSQL_LOG" instrument has been removed
- binlog io is now always instrumented with "wait/io/file/sql/binlog"
- the slow log is instrumented with a new name, "wait/io/file/sql/slow_log"
- the query log is instrumented with a new name, "wait/io/file/sql/query_log"
After the WL#2687, the binlog_cache_size and max_binlog_cache_size affect both the
stmt-cache and the trx-cache. This means that the resource used is twice the amount
expected/defined by the user.
The binlog_cache_use is incremented when the stmt-cache or the trx-cache is used
and binlog_cache_disk_use is incremented when the disk space from the stmt-cache or the
trx-cache is used. This behavior does not allow to distinguish which cache may be harming
performance due to the extra disk accesses and needs to have its in-memory cache
increased.
To fix the problem, we introduced two new options and status variables related to the
stmt-cache:
Options:
. binlog_stmt_cache_size
. max_binlog_stmt_cache_size
Status Variables:
. binlog_stmt_cache_use
. binlog_stmt_cache_disk_use
So there are
. binlog_cache_size that defines the size of the transactional cache for
updates to transactional engines for the binary log.
. binlog_stmt_cache_size that defines the size of the statement cache for
updates to non-transactional engines for the binary log.
. max_binlog_cache_size that sets the total size of the transactional
cache.
. max_binlog_stmt_cache_size that sets the total size of the statement
cache.
. binlog_cache_use that identifies the number of transactions that used the
temporary transactional binary log cache.
. binlog_cache_disk_use that identifies the number of transactions that used
the temporary transactional binary log cache but that exceeded the value of
binlog_cache_size.
. binlog_stmt_cache_use that identifies the number of statements that used the
temporary non-transactional binary log cache.
. binlog_stmt_cache_disk_use that identifies the number of statements that used
the temporary non-transactional binary log cache but that exceeded the value of
binlog_stmt_cache_size.
include/my_sys.h:
Updated message on disk_writes' usage.
mysql-test/extra/binlog_tests/binlog_cache_stat.test:
Updated the test case and added code to check the new status variables
binlog_stmt_cache_use and binlog_stmt_cache_disk_use.
mysql-test/extra/rpl_tests/rpl_binlog_max_cache_size.test:
Updated the test case to use the new system variables max_binlog_stmt_cache_size and binlog_stmt_cache_size.
mysql-test/r/mysqld--help-notwin.result:
Updated the result file.
mysql-test/suite/binlog/r/binlog_mixed_cache_stat.result:
Updated the result file.
mysql-test/suite/binlog/r/binlog_row_cache_stat.result:
Updated the result file.
mysql-test/suite/binlog/r/binlog_stm_cache_stat.result:
Updated the result file.
mysql-test/suite/rpl/r/rpl_mixed_binlog_max_cache_size.result:
Update the result file.
mysql-test/suite/rpl/r/rpl_row_binlog_max_cache_size.result:
Update the result file.
mysql-test/suite/rpl/r/rpl_stm_binlog_max_cache_size.result:
Updated the result file.
mysql-test/suite/sys_vars/inc/binlog_stmt_cache_size_basic.inc:
Added a test case to check the binlog_stmt_cache_size.
mysql-test/suite/sys_vars/r/binlog_stmt_cache_size_basic_32.result:
Updated the result file.
mysql-test/suite/sys_vars/r/binlog_stmt_cache_size_basic_64.result:
Updated the result file.
mysql-test/suite/sys_vars/r/max_binlog_stmt_cache_size_basic.result:
Updated the result file.
mysql-test/suite/sys_vars/t/binlog_stmt_cache_size_basic_32.test:
Added a test case to check the binlog_stmt_cache_size.
mysql-test/suite/sys_vars/t/binlog_stmt_cache_size_basic_64.test:
Added a test case to check the binlog_stmt_cache_size.
mysql-test/suite/sys_vars/t/max_binlog_cache_size_func-master.opt:
Removed because there is no test case max_binlog_cache_size_func.
mysql-test/suite/sys_vars/t/max_binlog_stmt_cache_size_basic.test:
Added a test case to check the system variable max_binlog_stmt_cache_size.
sql/log.cc:
There two main changes in here:
. Changed the set_write_error() as an error message is set according
to the type of the cache.
. Created the function set_binlog_cache_info where references to the
appropriate status and system variables are set and the server can
smoothly compute statistics and set the maximum size for each cache.
sql/log.h:
Changed the signature of the function in order to identify the error message
to be printed out as there is a different error code for each type of cache.
sql/mysqld.cc:
Added new status variables binlog_stmt_cache_use and binlog_stmt_cache_disk_use.
sql/mysqld.h:
Added new system variables max_binlog_stmt_cache_size and binlog_stmt_cache_size.
sql/share/errmsg-utf8.txt:
Added new error message related to the statement cache.
sql/sys_vars.cc:
Added new system variables max_binlog_stmt_cache_size and binlog_stmt_cache_size.
In sql/log.c, member function wait_for_update_bin_log, a condition is entered with
THD::enter_cond but is not exited. This might leave dangling references to the
mutex/condition in the per-thread information area.
To fix the problem, we call exit_cond to properly remove references to the mutex,
LOCK_log.
A CREATE...SELECT that fails is written to the binary log if a non-transactional
statement is updated. If the logging format is ROW, the CREATE statement and the
changes are written to the binary log as distinct events and by consequence the
created table is not rolled back in the slave.
In this patch, we opted to let the slave goes out of sync by not writting to the
binary log the CREATE statement. We do this by simply reseting the binary log's
cache.
mysql-test/suite/rpl/r/rpl_drop.result:
Added a test case.
mysql-test/suite/rpl/t/rpl_drop.test:
Added a test case.
sql/log.cc:
Introduced a function to clean up the cache.
sql/log.h:
Introduced a function to clean up the cache.
sql/sql_insert.cc:
Cleaned up the binary log cache if a CREATE...SELECT fails.
BUG#54872 MBR: replication failure caused by using tmp table inside transaction
Changed criteria to classify a statement as unsafe in order to reduce the
number of spurious warnings. So a statement is classified as unsafe when
there is on-going transaction at any point of the execution if:
1. The mixed statement is about to update a transactional table and
a non-transactional table.
2. The mixed statement is about to update a temporary transactional
table and a non-transactional table.
3. The mixed statement is about to update a transactional table and
read from a non-transactional table.
4. The mixed statement is about to update a temporary transactional
table and read from a non-transactional table.
5. The mixed statement is about to update a non-transactional table
and read from a transactional table when the isolation level is
lower than repeatable read.
After updating a transactional table if:
6. The mixed statement is about to update a non-transactional table
and read from a temporary transactional table.
7. The mixed statement is about to update a non-transactional table
and read from a temporary transactional table.
8. The mixed statement is about to update a non-transactionala table
and read from a temporary non-transactional table.
9. The mixed statement is about to update a temporary non-transactional
table and update a non-transactional table.
10. The mixed statement is about to update a temporary non-transactional
table and read from a non-transactional table.
11. A statement is about to update a non-transactional table and the
option variables.binlog_direct_non_trans_update is OFF.
The reason for this is that locks acquired may not protected a concurrent
transaction of interfering in the current execution and by consequence in
the result. So the patch reduced the number of spurious unsafe warnings.
Besides we fixed a regression caused by BUG#51894, which makes temporary
tables to go into the trx-cache if there is an on-going transaction. In
MIXED mode, the patch for BUG#51894 ignores that the trx-cache may have
updates to temporary non-transactional tables that must be written to the
binary log while rolling back the transaction.
So we fix this problem by writing the content of the trx-cache to the
binary log while rolling back a transaction if a non-transactional
temporary table was updated and the binary logging format is MIXED.
Conflicts:
Text conflict in mysql-test/r/archive.result
Contents conflict in mysql-test/r/innodb_bug38231.result
Text conflict in mysql-test/r/mdl_sync.result
Text conflict in mysql-test/suite/binlog/t/disabled.def
Text conflict in mysql-test/suite/rpl_ndb/r/rpl_ndb_binlog_format_errors.result
Text conflict in mysql-test/t/archive.test
Contents conflict in mysql-test/t/innodb_bug38231.test
Text conflict in mysql-test/t/mdl_sync.test
Text conflict in sql/sp_head.cc
Text conflict in sql/sql_show.cc
Text conflict in sql/table.cc
Text conflict in sql/table.h
This patch fixes two problems described as follows:
1 - If there is an on-going transaction and a temporary table is created or
dropped, any failed statement that follows the "create" or "drop commands"
triggers a rollback and by consequence the slave will go out sync because
the binary log will have a wrong sequence of events.
To fix the problem, we changed the expression that evaluates when the
cache should be flushed after either the rollback of a statment or
transaction.
2 - When a "CREATE TEMPORARY TABLE SELECT * FROM" was executed the
OPTION_KEEP_LOG was not set into the thd->options. For that reason, if
the transaction had updated only transactional engines and was rolled
back at the end (.e.g due to a deadlock) the changes were not written
to the binary log, including the creation of the temporary table.
To fix the problem, we have set the OPTION_KEEP_LOG into the thd->options
when a "CREATE TEMPORARY TABLE SELECT * FROM" is executed.
sql/log.cc:
Reorganized the code based on the following functions:
- bool ending_trans(const THD* thd, const bool all);
- bool trans_has_updated_non_trans_table(const THD* thd);
- bool trans_has_no_stmt_committed(const THD* thd, const bool all);
- bool stmt_has_updated_non_trans_table(const THD* thd);
sql/log.h:
Added functions to organize the code in log.cc.
sql/log_event.cc:
Removed the OPTION_KEEP_LOG since it must be used only when
creating and dropping temporary tables.
sql/log_event_old.cc:
Removed the OPTION_KEEP_LOG since it must be used only when
creating and dropping temporary tables.
sql/sql_parse.cc:
When a "CREATE TEMPORARY TABLE SELECT * FROM" was executed the
OPTION_KEEP_LOG was not set into the thd->options.
To fix the problem, we have set the OPTION_KEEP_LOG into the
thd->options when a "CREATE TEMPORARY TABLE SELECT * FROM"
is executed.
MYSQL_BIN_LOG m_table_map_version member and it's associated
functions were not used in the logic of binlogging and replication,
this patch removed all related code.
sql/log.cc:
removed unused m_table_map_version variable and functions
sql/log.h:
removed unused m_table_map_version variable and functions
sql/log_event.h:
Removed unused LOG_EVENT_UPDATE_TABLE_MAP_VERSION_F flag
sql/sql_class.cc:
Removed unused LOG_EVENT_UPDATE_TABLE_MAP_VERSION_F flag
sql/sql_load.cc:
Removed unused LOG_EVENT_UPDATE_TABLE_MAP_VERSION_F flag
sql/table.cc:
removed unused table_map_version variable
sql/table.h:
removed unused table_map_version variable
transaction
BUG#52616 Temp table prevents switch binlog format from STATEMENT to ROW
Before the WL#2687 and BUG#46364, every non-transactional change that happened
after a transactional change was written to trx-cache and flushed upon
committing the transaction. WL#2687 and BUG#46364 changed this behavior and
non-transactional changes are now written to the binary log upon committing
the statement.
A binary log event is identified as transactional or non-transactional through
a flag in the Log_event which is set taking into account the underlie storage
engine on what it is stems from. In the current bug, this flag was not being
set properly when the DROP TEMPORARY TABLE was executed.
However, while fixing this bug we figured out that changes to temporary tables
should be always written to the trx-cache if there is an on-going transaction.
Otherwise, binlog events in the reversed order would be produced.
Regarding concurrency, keeping changes to temporary tables in the trx-cache is
also safe as temporary tables are only visible to the owner connection.
In this patch, we classify the following statements as unsafe:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
On the other hand, the following statements are classified as safe:
1 - INSERT INTO t_innodb SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_innodb
The patch also guarantees that transactions that have a DROP TEMPORARY are
always written to the binary log regardless of the mode and the outcome:
commit or rollback. In particular, the DROP TEMPORARY is extended with the
IF EXISTS clause when the current statement logging format is set to row.
Finally, the patch allows to switch from STATEMENT to MIXED/ROW when there
are temporary tables but the contrary is not possible.
mysql-test/extra/rpl_tests/rpl_binlog_max_cache_size.test:
Updated the test case because
CREATE TEMPORARY TABLE t_innodb_temp SELECT * FROM t_myisam is not unsafe.
mysql-test/extra/rpl_tests/rpl_implicit_commit_binlog.test:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/extra/rpl_tests/rpl_innodb.test:
Removed comments from the test case that became false after the patch.
mysql-test/extra/rpl_tests/rpl_loaddata.test:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
mysql-test/include/ctype_utf8_table.inc:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
mysql-test/r/ctype_cp932_binlog_stm.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/binlog/r/binlog_database.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/binlog/r/binlog_innodb_row.result:
Updated the result file.
mysql-test/suite/binlog/r/binlog_multi_engine.result:
Updated the unsafe message.
mysql-test/suite/binlog/r/binlog_row_binlog.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/binlog/r/binlog_row_drop_tmp_tbl.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/binlog/r/binlog_stm_binlog.result:
Updated the result file.
mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/binlog/t/binlog_tmp_table.test:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
mysql-test/suite/ndb/r/ndb_binlog_format.result:
Updated the unsafe message.
mysql-test/suite/rpl/r/rpl_concurrency_error.result:
Updated the unsafe message.
mysql-test/suite/rpl/r/rpl_mixed_binlog_max_cache_size.result:
Updated the result file because
CREATE TEMPORARY TABLE t_innodb_temp SELECT * FROM t_myisam is not unsafe.
mysql-test/suite/rpl/r/rpl_mixed_implicit_commit_binlog.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_mixed_mixing_engines.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_mixed_row_innodb.result:
Added some comments to ease the understanding of the result file.
mysql-test/suite/rpl/r/rpl_non_direct_mixed_mixing_engines.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_non_direct_row_mixing_engines.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_non_direct_stm_mixing_engines.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_row_drop.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_row_implicit_commit_binlog.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_row_mixing_engines.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_stm_binlog_max_cache_size.result:
Updated the result file because
CREATE TEMPORARY TABLE t_innodb_temp SELECT * FROM t_myisam is not unsafe.
mysql-test/suite/rpl/r/rpl_stm_implicit_commit_binlog.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_stm_innodb.result:
Added some comments to ease the understanding of the result file.
mysql-test/suite/rpl/r/rpl_stm_mixing_engines.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl/r/rpl_stm_stop_middle_group.result:
Updated the unsafe message.
mysql-test/suite/rpl/r/rpl_temp_temporary.result:
Added a test case.
mysql-test/suite/rpl/t/rpl000013.test:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
mysql-test/suite/rpl/t/rpl_misc_functions.test:
Suppressed warning messages.
mysql-test/suite/rpl/t/rpl_temp_table.test:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
mysql-test/suite/rpl/t/rpl_temp_temporary.test:
Added a test case.
mysql-test/suite/rpl/t/rpl_temporary.test:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
mysql-test/suite/rpl_ndb/r/rpl_ndb_row_implicit_commit_binlog.result:
Updated the test case due to the new rules: changes to
temporary tables are written to the binary log in the
boundaries of a transaction if there is any.
mysql-test/suite/rpl_ndb/r/rpl_truncate_7ndb.result:
Updated the test case to remove references to positions
in the binary log.
mysql-test/suite/rpl_ndb/t/rpl_truncate_7ndb.test:
Updated the test case to remove references to positions
in the binary log.
mysql-test/t/create_select_tmp.test:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
mysql-test/t/ctype_cp932_binlog_stm.test:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
mysql-test/t/mysqlbinlog.test:
Suppressed warning messages due to the following cases:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
sql/log.cc:
Improved the code by creating several functions to hide decision
on type of engine changed, commit/abort, etc:
. stmt_has_updated_non_trans_table
. trans_has_updated_non_trans_table
. ending_trans
Updated the binlog_rollback function and the use of the
OPTION_KEEP_LOG which indincates when a temporary table was
either created or dropped and as such the command must be
logged if not in MIXED mode and even while rolling back the
transaction.
sql/log.h:
Improved the code by creating several functions to hide decision
on type of engine changed, commit/abort, etc.
sql/log_event.cc:
Removed the setting of the OPTION_KEEP_LOG as it is related to CREATE
TEMPORARY and DROP TEMPORARY and not to the type of engine (i.e.
transactional or non-transactional).
sql/log_event_old.cc:
Removed the setting of the OPTION_KEEP_LOG as it is related to CREATE
TEMPORARY and DROP TEMPORARY and not to the type of engine (i.e.
transactional or non-transactional).
sql/share/errmsg-utf8.txt:
Updated the unsafe message.
sql/sql_class.cc:
Classifies the following statements as unsafe:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
On the other hand, the following statements are classified as safe:
1 - INSERT INTO t_innodb SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_innodb
sql/sql_class.h:
It allows to switch from STATEMENT to MIXED/ROW when there are temporary
tables but the contrary is not possible.
sql/sql_table.cc:
Fixed the case that a DROP/DROP TEMPORARY that affects a temporary table in MIXED
mode is written as a DROP TEMPORARY TABLE IF EXISTS because the table may not exist in the slave and due to the IF EXISTS token an error will never happen
while processing the statement in the slave.
Removed a function that was not being used.
This patch:
- Moves all definitions from the mysql_priv.h file into
header files for the component where the variable is
defined
- Creates header files if the component lacks one
- Eliminates all include directives from mysql_priv.h
- Eliminates all circular include cycles
- Rename time.cc to sql_time.cc
- Rename mysql_priv.h to sql_priv.h
Conflicts:
- mysql-test/r/mysqld--help-win.result
- sql/sys_vars.cc
Original revsion (in next-mr-bugfixing):
------------------------------------------------------------
revno: 2971 [merge]
revision-id: alfranio.correia@sun.com-20100121210527-rbuheu5rnsmcakh1
committer: Alfranio Correia <alfranio.correia@sun.com>
branch nick: mysql-next-mr-bugfixing
timestamp: Thu 2010-01-21 21:05:27 +0000
message:
BUG#46364 MyISAM transbuffer problems (NTM problem)
It is well-known that due to concurrency issues, a slave can become
inconsistent when a transaction contains updates to both transaction and
non-transactional tables.
In a nutshell, the current code-base tries to preserve causality among the
statements by writing non-transactional statements to the txn-cache which
is flushed upon commit. However, modifications done to non-transactional
tables on behalf of a transaction become immediately visible to other
connections but may not immediately get into the binary log and therefore
consistency may be broken.
In general, it is impossible to automatically detect causality/dependency
among statements by just analyzing the statements sent to the server. This
happen because dependency may be hidden in the application code and it is
necessary to know a priori all the statements processed in the context of
a transaction such as in a procedure. Moreover, even for the few cases that
we could automatically address in the server, the computation effort
required could make the approach infeasible.
So, in this patch we introduce the option
- "--binlog-direct-non-transactional-updates" that can be used to bypass
the current behavior in order to write directly to binary log statements
that change non-transactional tables.
Besides, it is used to enable the WL#2687 which is disabled by default.
------------------------------------------------------------
revno: 2970.1.1
revision-id: alfranio.correia@sun.com-20100121131034-183r4qdyld7an5a0
parent: alik@sun.com-20100121083914-r9rz2myto3tkdya0
committer: Alfranio Correia <alfranio.correia@sun.com>
branch nick: mysql-next-mr-bugfixing
timestamp: Thu 2010-01-21 13:10:34 +0000
message:
BUG#46364 MyISAM transbuffer problems (NTM problem)
It is well-known that due to concurrency issues, a slave can become
inconsistent when a transaction contains updates to both transaction and
non-transactional tables.
In a nutshell, the current code-base tries to preserve causality among the
statements by writing non-transactional statements to the txn-cache which
is flushed upon commit. However, modifications done to non-transactional
tables on behalf of a transaction become immediately visible to other
connections but may not immediately get into the binary log and therefore
consistency may be broken.
In general, it is impossible to automatically detect causality/dependency
among statements by just analyzing the statements sent to the server. This
happen because dependency may be hidden in the application code and it is
necessary to know a priori all the statements processed in the context of
a transaction such as in a procedure. Moreover, even for the few cases that
we could automatically address in the server, the computation effort
required could make the approach infeasible.
So, in this patch we introduce the option
- "--binlog-direct-non-transactional-updates" that can be used to bypass
the current behavior in order to write directly to binary log statements
that change non-transactional tables.
Besides, it is used to enable the WL#2687 which is disabled by default.
Conflicts:
Text conflict in .bzr-mysql/default.conf
Text conflict in mysql-test/extra/rpl_tests/rpl_loaddata.test
Text conflict in mysql-test/r/mysqlbinlog2.result
Text conflict in mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result
Text conflict in mysql-test/suite/binlog/r/binlog_unsafe.result
Text conflict in mysql-test/suite/rpl/r/rpl_insert_id.result
Text conflict in mysql-test/suite/rpl/r/rpl_loaddata.result
Text conflict in mysql-test/suite/rpl/r/rpl_stm_auto_increment_bug33029.result
Text conflict in mysql-test/suite/rpl/r/rpl_udf.result
Text conflict in mysql-test/suite/rpl/t/rpl_slow_query_log.test
Text conflict in sql/field.h
Text conflict in sql/log.cc
Text conflict in sql/log_event.cc
Text conflict in sql/log_event_old.cc
Text conflict in sql/mysql_priv.h
Text conflict in sql/share/errmsg.txt
Text conflict in sql/sp.cc
Text conflict in sql/sql_acl.cc
Text conflict in sql/sql_base.cc
Text conflict in sql/sql_class.h
Text conflict in sql/sql_db.cc
Text conflict in sql/sql_delete.cc
Text conflict in sql/sql_insert.cc
Text conflict in sql/sql_lex.cc
Text conflict in sql/sql_lex.h
Text conflict in sql/sql_load.cc
Text conflict in sql/sql_table.cc
Text conflict in sql/sql_update.cc
Text conflict in sql/sql_view.cc
Conflict adding files to storage/innobase. Created directory.
Conflict because storage/innobase is not versioned, but has versioned children. Versioned directory.
Conflict adding file storage/innobase. Moved existing file to storage/innobase.moved.
Conflict adding files to storage/innobase/handler. Created directory.
Conflict because storage/innobase/handler is not versioned, but has versioned children. Versioned directory.
Contents conflict in storage/innobase/handler/ha_innodb.cc
Bug#16565 mysqld --help --verbose does not order variablesBug#20413 sql_slave_skip_counter is not shown in show variables
Bug#20415 Output of mysqld --help --verbose is incomplete
Bug#25430 variable not found in SELECT @@global.ft_max_word_len;
Bug#32902 plugin variables don't know their names
Bug#34599 MySQLD Option and Variable Reference need to be consistent in formatting!
Bug#34829 No default value for variable and setting default does not raise error
Bug#34834 ? Is accepted as a valid sql mode
Bug#34878 Few variables have default value according to documentation but error occurs
Bug#34883 ft_boolean_syntax cant be assigned from user variable to global var.
Bug#37187 `INFORMATION_SCHEMA`.`GLOBAL_VARIABLES`: inconsistent status
Bug#40988 log_output_basic.test succeeded though syntactically false.
Bug#41010 enum-style command-line options are not honoured (maria.maria-recover fails)
Bug#42103 Setting key_buffer_size to a negative value may lead to very large allocations
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
Bug#44797 plugins w/o command-line options have no disabling option in --help
Bug#46314 string system variables don't support expressions
Bug#46470 sys_vars.max_binlog_cache_size_basic_32 is broken
Bug#46586 When using the plugin interface the type "set" for options caused a crash.
Bug#47212 Crash in DBUG_PRINT in mysqltest.cc when trying to print octal number
Bug#48758 mysqltest crashes on sys_vars.collation_server_basic in gcov builds
Bug#49417 some complaints about mysqld --help --verbose output
Bug#49540 DEFAULT value of binlog_format isn't the default value
Bug#49640 ambiguous option '--skip-skip-myisam' (double skip prefix)
Bug#49644 init_connect and \0
Bug#49645 init_slave and multi-byte characters
Bug#49646 mysql --show-warnings crashes when server dies
CMakeLists.txt:
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
client/mysql.cc:
don't crash with --show-warnings when mysqld dies
config/ac-macros/plugins.m4:
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
include/my_getopt.h:
comments
include/my_pthread.h:
fix double #define
mysql-test/mysql-test-run.pl:
run sys_vars suite by default
properly recognize envirinment variables (e.g. MTR_MAX_SAVE_CORE) set to 0
escape gdb command line arguments
mysql-test/suite/sys_vars/r/rpl_init_slave_func.result:
init_slave+utf8 bug
mysql-test/suite/sys_vars/t/rpl_init_slave_func.test:
init_slave+utf8 bug
mysys/my_getopt.c:
Bug#34599 MySQLD Option and Variable Reference need to be consistent in formatting!
Bug#46586 When using the plugin interface the type "set" for options caused a crash.
Bug#49640 ambiguous option '--skip-skip-myisam' (double skip prefix)
mysys/typelib.c:
support for flagset
sql/ha_ndbcluster.cc:
backport from telco tree
sql/item_func.cc:
Bug#49644 init_connect and \0
Bug#49645 init_slave and multi-byte characters
sql/sql_builtin.cc.in:
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
sql/sql_plugin.cc:
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
Bug#32902 plugin variables don't know their names
Bug#44797 plugins w/o command-line options have no disabling option in --help
sql/sys_vars.cc:
all server variables are defined here
storage/myisam/ft_parser.c:
remove unnecessary updates of param->quot
storage/myisam/ha_myisam.cc:
myisam_* variables belong here
strings/my_vsnprintf.c:
%o and %llx
unittest/mysys/my_vsnprintf-t.c:
%o and %llx tests
vio/viosocket.c:
bugfix: fix @@wait_timeout to work with socket timeouts (vs. alarm thread)
This patch fixes three bugs as follows. First, aborting the server while purging
binary logs might generate orphan files due to how the purge operation was
implemented:
(purge routine - sql/log.cc - MYSQL_BIN_LOG::purge_logs)
1 - register the files to be removed in a temporary buffer.
2 - update the log-bin.index.
3 - flush the log-bin.index.
4 - erase the files whose names where register in the temporary buffer
in step 1.
Thus a failure while executing step 4 would generate an orphan file. Second,
a similar issue might happen while creating a new binary as follows:
(create routine - sql/log.cc - MYSQL_BIN_LOG::open)
1 - open the new log-bin.
2 - update the log-bin.index.
Thus a failure while executing step 1 would generate an orphan file.
To fix these issues, we record the files to be purged or created before really
removing or adding them. So if a failure happens such records can be used to
automatically remove dangling files. The new steps might be outlined as follows:
(purge routine - sql/log.cc - MYSQL_BIN_LOG::purge_logs)
1 - register the files to be removed in the log-bin.~rec~ placed in
the data directory.
2 - update the log-bin.index.
3 - flush the log-bin.index.
4 - delete the log-bin.~rec~.
(create routine - sql/log.cc - MYSQL_BIN_LOG::open)
1 - register the file to be created in the log-bin.~rec~
placed in the data directory.
2 - open the new log-bin.
3 - update the log-bin.index.
4 - delete the log-bin.~rec~.
(recovery routine - sql/log.cc - MYSQL_BIN_LOG::open_index_file)
1 - open the log-bin.index.
2 - open the log-bin.~rec~.
3 - for each file in log-bin.~rec~.
3.1 Check if the file is in the log-bin.index and if so ignore it.
3.2 Otherwise, delete it.
The third issue can be described as follows. The purge operation was allowing
to remove a file in use thus leading to the loss of data and possible
inconsistencies between the master and slave. Roughly, the routine was only
taking into account the dump threads and so if a slave was not connect the
file might be delete even though it was in use.
Support for flushing individual logs, so that the user can
selectively flush a subset of the server logs.
Flush of individual logs is done according to the
following syntax:
FLUSH <log_category> LOGS;
The syntax is extended so that the user is able to flush a
subset of logs:
FLUSH [log_category LOGS,];
where log_category is one of:
SLOW
ERROR
BINARY
ENGINE
GENERAL
RELAY.
mysql-test/suite/rpl/r/rpl_flush_logs.result:
Test result for WL#5142.
mysql-test/suite/rpl/t/rpl_flush_logs.test:
Added the test file to verify if the 'flush individual log'
statement works fine.
sql/log.cc:
Added the two functions to flush slow and general log.
sql/sql_parse.cc:
Added code to flush specified logs against the option.
sql/sql_yacc.yy:
Added code to parse the 'flush * log' statement syntax and
set its option to Lex->type.
beyond unsigned long.
BUG#44779: binlog.binlog_max_extension may be causing failure on
next test in PB
NOTE1: this is the backport to next-mr.
NOTE2: already includes patch for BUG#44779.
Binlog file extensions would turn into negative numbers once the
variable used to hold the value reached maximum for signed
long. Consequently, incrementing value to the next (negative) number
would lead to .000000 extension, causing the server to fail.
This patch addresses this issue by not allowing negative extensions
and by returning an error on find_uniq_filename, when the limit is
reached. Additionally, warnings are printed to the error log when the
limit is approaching. FLUSH LOGS will also report warnings to the
user, if the extension number has reached the limit. The limit has been
set to 0x7FFFFFFF as the maximum.
mysql-test/suite/binlog/t/binlog_max_extension.test:
Test case added that checks the maximum available number for
binlog extensions.
sql/log.cc:
Changes to find_uniq_filename and test_if_number.
sql/log.h:
Added macros with values for MAX_LOG_UNIQUE_FN_EXT and
LOG_WARN_UNIQUE_FN_EXT_LEFT, as suggested in review.
The mentioned on the bug report set of bugs fixes have not be pushed to the main trees.
Fixed with extracting commits done to 6.0-rpl tree and applying them to the main 5.1.
Notes.
1. part of changes - the mtr's specific - were packported to the main 5.0 tree for mtr v1
as http://lists.mysql.com/commits/46562
However, there is no that fix anymore in the mtr v2. (This fact was mailed to mtr maintaining
people).
2. Bug@36929 crash in kill_zombie_dump_threads-> THD::awake() with replication tests
is not backported because the base code of the patch is libevent and that was removed
from the main trees due to its instability.
client/mysqlbinlog.cc:
fixes for BUG#35546
mysql-test/suite/rpl/r/rpl_bug41902.result:
the new tests result file is added.
mysql-test/suite/rpl/t/rpl_bug41902-slave.opt:
conf file for bug41902 testing is added.
mysql-test/suite/rpl/t/rpl_bug41902.test:
regression tests for Bug #41902 is added.
sql/log.cc:
collection of changes due to Bug #48463.
sql/log.h:
collection of changes due to Bug #48463.
sql/rpl_rli.h:
collection of changes due to Bug #48463.
sql/slave.cc:
collection of changes due to Bug #48463.
sql/sql_repl.cc:
collection of changes due to Bug #48463.
Non-transactional updates that take place inside a transaction present problems
for logging because they are visible to other clients before the transaction
is committed, and they are not rolled back even if the transaction is rolled
back. It is not always possible to log correctly in statement format when both
transactional and non-transactional tables are used in the same transaction.
In the current patch, we ensure that such scenario is completely safe under the
ROW and MIXED modes.
BUG#31665 sync_binlog should cause relay logs to be synchronized
NOTE: Backporting the patch to next-mr.
Add sync_relay_log option to server, this option works for relay log
the same as option sync_binlog for binlog. This option also synchronize
master info to disk when set to non-zero value.
Original patches from Sinisa and Mark, with some modifications
