The test was used to fail because of
UPDATE t3,t4 SET t3.a=t4.a + bug27417(1);
did not prescribe the order of two row operations implied by the update.
Fixed with forcing the order with adding a where condition w/o
affecting the former bug fixes logics.
Clarified error messages related to unsafe statements:
- avoid the internal technical term "row injection"
- use 'binary log' instead of 'binlog'
- avoid the word 'unsafeness'
parallel mode
The failure has nothing to do with parallel, but rather on the
order the tests are executed. In this case, the test
binlog_tmp_table (lets call it test2) was not ensuring that the
binary logs would be reset when it started. Later the test issues
a mysqlbinlog .../master-bin.000002 | mysql ... If the test that
was executed before this one (lets call it test1) had issued a
flush logs, then the file in use in test1 (master-bin.000002)
would not actually match the one that was expected. Eventually,
this would cause the statements logged in test1 to be replayed,
instead of the ones logged in the beginning of test2.
We fix this by:
1. adding RESET MASTER to the beginning of binlog_tmp_table
2. setting dynamically the file to use in binlog_tmp_table
Only #1 was needed, but the two make the tests cases more robust.
transaction
BUG#52616 Temp table prevents switch binlog format from STATEMENT to ROW
Before the WL#2687 and BUG#46364, every non-transactional change that happened
after a transactional change was written to trx-cache and flushed upon
committing the transaction. WL#2687 and BUG#46364 changed this behavior and
non-transactional changes are now written to the binary log upon committing
the statement.
A binary log event is identified as transactional or non-transactional through
a flag in the Log_event which is set taking into account the underlie storage
engine on what it is stems from. In the current bug, this flag was not being
set properly when the DROP TEMPORARY TABLE was executed.
However, while fixing this bug we figured out that changes to temporary tables
should be always written to the trx-cache if there is an on-going transaction.
Otherwise, binlog events in the reversed order would be produced.
Regarding concurrency, keeping changes to temporary tables in the trx-cache is
also safe as temporary tables are only visible to the owner connection.
In this patch, we classify the following statements as unsafe:
1 - INSERT INTO t_myisam SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_myisam
3 - CREATE TEMPORARY TABLE t_myisam_temp SELECT * FROM t_myisam
On the other hand, the following statements are classified as safe:
1 - INSERT INTO t_innodb SELECT * FROM t_myisam_temp
2 - INSERT INTO t_myisam_temp SELECT * FROM t_innodb
The patch also guarantees that transactions that have a DROP TEMPORARY are
always written to the binary log regardless of the mode and the outcome:
commit or rollback. In particular, the DROP TEMPORARY is extended with the
IF EXISTS clause when the current statement logging format is set to row.
Finally, the patch allows to switch from STATEMENT to MIXED/ROW when there
are temporary tables but the contrary is not possible.
Conflicts:
Text conflict in mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result
Text conflict in sql/log.cc
Text conflict in sql/set_var.cc
Text conflict in sql/sql_class.cc
BUG#46364 introduced the flag binlog_direct_non_transactional_updates which
would make N-changes to be written to the binary log upon committing the
statement when "ON". On the other hand, when "OFF" the option was supposed
to mimic the behavior in 5.1. However, the implementation was not mimicking
the behavior correctly and the following bugs popped up:
Case #1: N-changes executed within a transaction would go into
the S-cache. When later in the same transaction a
T-change occurs, N-changes following it were written
to the T-cache instead of the S-cache. In some cases,
this raises problems. For example, a
Table_map_log_event being written initially into the
S-cache, together with the initial N-changes, would be
absent from the T-cache. This would log N-changes
orphaned from a Table_map_log_event (thence discarded
at the slave). (MIXED and ROW)
Case #2: When rolling back a transaction, the N-changes that
might be in the T-cache were disregarded and
truncated along with the T-changes. (MIXED and ROW)
Case #3: When a MIXED statement (TN) is ahead of any other
T-changes in the transaction and it fails, it is kept
in the T-cache until the transaction ends. This is
not the case in 5.1 or Betony (5.5.2). In these, the
failed TN statement would be written to the binlog at
the same instant it had failed and not deferred until
transaction end. (SBR)
To fix these problems, we have decided to do what follows:
For Case #1 and #2, we circumvent them:
1. by not letting binlog_direct_non_transactional_updates
affect MIXED and RBR. These modes will keep the behavior
provided by WL#2687. Although this will make Celosia to
behave differently from 5.1, an execution will be always
safe under such modes in the sense that slaves will never
go out sync. In 5.1, using either MIXED or ROW while
mixing N-statements and T-statements was not safe.
For Case #3, we don't actually fix it. We:
1. keep it and make all MIXED statements whether they end
up failing or not or whether they are up front in the
transaction or after some transactional change to always
be stored in the T-cache. This means that it is written
to the binary log on transaction commit/rollback only.
2. We make the warning message even more specific about the
MIXED statement and SBR.
Conflicts:
Text conflict in client/mysqlbinlog.cc
Text conflict in mysql-test/Makefile.am
Text conflict in mysql-test/collections/default.daily
Text conflict in mysql-test/r/mysqlbinlog_row_innodb.result
Text conflict in mysql-test/suite/rpl/r/rpl_typeconv_innodb.result
Text conflict in mysql-test/suite/rpl/t/rpl_get_master_version_and_clock.test
Text conflict in mysql-test/suite/rpl/t/rpl_row_create_table.test
Text conflict in mysql-test/suite/rpl/t/rpl_slave_skip.test
Text conflict in mysql-test/suite/rpl/t/rpl_typeconv_innodb.test
Text conflict in mysys/charset.c
Text conflict in sql/field.cc
Text conflict in sql/field.h
Text conflict in sql/item.h
Text conflict in sql/item_func.cc
Text conflict in sql/log.cc
Text conflict in sql/log_event.cc
Text conflict in sql/log_event_old.cc
Text conflict in sql/mysqld.cc
Text conflict in sql/rpl_utility.cc
Text conflict in sql/rpl_utility.h
Text conflict in sql/set_var.cc
Text conflict in sql/share/Makefile.am
Text conflict in sql/sql_delete.cc
Text conflict in sql/sql_plugin.cc
Text conflict in sql/sql_select.cc
Text conflict in sql/sql_table.cc
Text conflict in storage/example/ha_example.h
Text conflict in storage/federated/ha_federated.cc
Text conflict in storage/myisammrg/ha_myisammrg.cc
Text conflict in storage/myisammrg/myrg_open.c
When mysqlbinlog was given the --database=X flag, it always printed
'ROLLBACK TO', but the corresponding 'SAVEPOINT' statement was not
printed. The replicated filter(replicated-do/ignore-db) and binlog
filter (binlog-do/ignore-db) has the same problem. They are solved
in this patch together.
After this patch, We always check whether the query is 'SAVEPOINT'
statement or not. Because this is a literal check, 'SAVEPOINT' and
'ROLLBACK TO' statements are also binlogged in uppercase with no
any comments.
The binlog before this patch can be handled correctly except one case
that any comments are in front of the keywords. for example:
/* bla bla */ SAVEPOINT a;
/* bla bla */ ROLLBACK TO a;
Conflicts:
Text conflict in client/mysqlbinlog.cc
Text conflict in mysql-test/r/explain.result
Text conflict in mysql-test/r/subselect.result
Text conflict in mysql-test/r/subselect3.result
Text conflict in mysql-test/r/type_datetime.result
Text conflict in sql/share/Makefile.am
The test case added in previous patch missed a RESET MASTER on
test start up. Without it, showing binary log contents can
sometimes show spurious entries from previously executed tests,
ultimately causing test failure - result mismatch.
The test file was added in:
revid:luis.soares@sun.com-20100224190153-k0bpdx9abe88uoo2
This patch also moves the test case into binlog_innodb_row.test
file. This way we avoid having yet another test file,
binlog_row_innodb_truncate.test, whose only purpose is to host
one test case. This had been actually suggested during original
patch review, but somehow the binlog_innodb_row was missed when
searching for a file to host the test case.
+ failing statements
Implicit DROP event for temporary table is not getting
LOG_EVENT_THREAD_SPECIFIC_F flag, because, in the previous
executed statement in the same thread, which might even be a
failed statement, the thread_specific_used flag is set to
FALSE (in mysql_reset_thd_for_next_command) and not set to TRUE
before connection is shutdown. This means that implicit DROP
event will take the FALSE value from thread_specific_used and
will not set LOG_EVENT_THREAD_SPECIFIC_F in the event header. As
a consequence, mysqlbinlog will not print the pseudo_thread_id
from the DROP event, because one of the requirements for the
printout is that this flag is set to TRUE.
We fix this by setting thread_specific_used whenever we are
binlogging a DROP in close_temporary_tables, and resetting it to
its previous value afterward.
For temporary tables that are created with an engine that does
not provide the HTON_CAN_RECREATE, the truncate operation is
performed resorting to the optimized handler::ha_delete_all_rows
method. However, this means that the truncate will share
execution path, from mysql_delete, with truncate on regular
tables and other delete operations. As a consequence the truncate
operation, for the temporary table is logged, even if in row mode
because there is no distinction between this and the other delete
operations at binlogging time.
We fix this by checking if: (i) the binlog format, when the
truncate operation was issued, is ROW; (ii) if the operation is a
truncate; and (iii) if the table is a temporary table; before
writing to the binary log. If all three conditions are met, we
skip writing to the binlog. A side effect of this fix is that we
limit the scope of setting and resetting the
current_stmt_binlog_row_based. Now we just set and reset it
inside mysql_delete in the boundaries of the
handler::ha_write_row loop. This way we have access to
thd->current_stmt_binlog_row_based real value inside
mysql_delete.
SUPER_ACL should be checked unconditionally while verifying if the binlog_format
or the binlog_direct_non_transactional_updates might be changed.
Roughly speaking, both session values cannot be changed in the context of a
transaction or a stored function. Note that changing the global value does
not cause any effect until a new connection is created.
So, we fixed the problem by first checking the permissions and right after further
verifications are ignored if the global value is being updated. In this patch, we
also re-structure the test case to make it more readable.
The task is to
(a) add a comment on indexes and
(b) increase the maximum length of column, table and the new index comments.
The patch committed on behalf of Yoshinori Matsunobu (Yoshinori.Matsunobu@Sun.COM).
1. Now test use fake_relay_log primitive
2. Added RESET SLAVE to include/setup_fake_relay_log.inc for removing relay log info file
3. Added RESET SLAVE to include/cleanup_fake_relay_log.inc
4. Test moved to rpl suite as rpl_binlog_auto_inc_bug33029.test
5. Updated result file
As part of BUG@39934 fix, the public:
- THD::current_stmt_binlog_row_based
variable had been removed and replaced by a private variable:
- THD::current_stmt_binlog_format.
THD was refactored and some modifiers and accessors were
implemented for the new variable.
However, due to a bad merge, the
THD::current_stmt_binlog_row_based variable is back as a public
member of THD. This in itself is already potentially
harmful. What's even worse is that while merging some more
patches and resolving conflicts, the variable started being used
again, which is obviously wrong.
To fix this we:
1. remove the extraneous variable from sql_class.h
2. revert a bad merge for BUG#49132
3. merge BUG#49132 properly again (actually, making use of the
cset used to merge the original patch to mysql-pe).
Cherry-pick a fix Bug#37148 from next-mr, to preserve
file ids of the added files, and ensure that all the necessary
changes have been pulled.
Since initially Bug#37148 was null-merged into 6.0,
the changeset that is now being cherry-picked was likewise
null merged into next-4284.
Now that Bug#37148 has been reapplied to 6.0, try to make
it work with next-4284. This is also necessary to be able
to pull other changes from 5.1-rep into next-4284.
To resolve the merge issues use this changeset applied
to 6.0:
revid:jperkin@sun.com-20091216103628-ylhqf7s6yegui2t9
revno: 3776.1.1
committer: He Zhenxing <zhenxing.he@sun.com>
branch nick: 6.0-codebase-bugfixing
timestamp: Thu 2009-12-17 17:02:50 +0800
message:
Fix merge problem with Bug#37148
Conflicts:
- mysql-test/r/mysqld--help-win.result
- sql/sys_vars.cc
Original revsion (in next-mr-bugfixing):
------------------------------------------------------------
revno: 2971 [merge]
revision-id: alfranio.correia@sun.com-20100121210527-rbuheu5rnsmcakh1
committer: Alfranio Correia <alfranio.correia@sun.com>
branch nick: mysql-next-mr-bugfixing
timestamp: Thu 2010-01-21 21:05:27 +0000
message:
BUG#46364 MyISAM transbuffer problems (NTM problem)
It is well-known that due to concurrency issues, a slave can become
inconsistent when a transaction contains updates to both transaction and
non-transactional tables.
In a nutshell, the current code-base tries to preserve causality among the
statements by writing non-transactional statements to the txn-cache which
is flushed upon commit. However, modifications done to non-transactional
tables on behalf of a transaction become immediately visible to other
connections but may not immediately get into the binary log and therefore
consistency may be broken.
In general, it is impossible to automatically detect causality/dependency
among statements by just analyzing the statements sent to the server. This
happen because dependency may be hidden in the application code and it is
necessary to know a priori all the statements processed in the context of
a transaction such as in a procedure. Moreover, even for the few cases that
we could automatically address in the server, the computation effort
required could make the approach infeasible.
So, in this patch we introduce the option
- "--binlog-direct-non-transactional-updates" that can be used to bypass
the current behavior in order to write directly to binary log statements
that change non-transactional tables.
Besides, it is used to enable the WL#2687 which is disabled by default.
------------------------------------------------------------
revno: 2970.1.1
revision-id: alfranio.correia@sun.com-20100121131034-183r4qdyld7an5a0
parent: alik@sun.com-20100121083914-r9rz2myto3tkdya0
committer: Alfranio Correia <alfranio.correia@sun.com>
branch nick: mysql-next-mr-bugfixing
timestamp: Thu 2010-01-21 13:10:34 +0000
message:
BUG#46364 MyISAM transbuffer problems (NTM problem)
It is well-known that due to concurrency issues, a slave can become
inconsistent when a transaction contains updates to both transaction and
non-transactional tables.
In a nutshell, the current code-base tries to preserve causality among the
statements by writing non-transactional statements to the txn-cache which
is flushed upon commit. However, modifications done to non-transactional
tables on behalf of a transaction become immediately visible to other
connections but may not immediately get into the binary log and therefore
consistency may be broken.
In general, it is impossible to automatically detect causality/dependency
among statements by just analyzing the statements sent to the server. This
happen because dependency may be hidden in the application code and it is
necessary to know a priori all the statements processed in the context of
a transaction such as in a procedure. Moreover, even for the few cases that
we could automatically address in the server, the computation effort
required could make the approach infeasible.
So, in this patch we introduce the option
- "--binlog-direct-non-transactional-updates" that can be used to bypass
the current behavior in order to write directly to binary log statements
that change non-transactional tables.
Besides, it is used to enable the WL#2687 which is disabled by default.
The auto-inc unsafe warning makes sense even though it's just
one auto-inc table could be involved via a trigger or a stored
function.
However its content was not updated by bug@45677 fixes continuing to mention
two tables whereas the fixes refined semantics of replication of auto_increment
in stored routine.
Fixed with updating the error message, renaming the error and an internal unsafe-condition
constants.
A documentation notice
======================
Inserting into an autoincrement column in a stored function or a trigger
is unsafe for replication.
Even with just one autoincrement column, if the routine is invoked more than
once slave is not guaranteed to execute the statement graph same way as
the master.
And since it's impossible to estimate how many times a routine can be invoked at
the query pre-execution phase (see lock_tables), the statement is marked
pessimistically unsafe.
Bug #45856 can't switch from binlog_format=row to mix with open tmp tbl
If binlog_format=MIXED, there are open temporary tables, an unsafe statement
is executed, and the user issues 'SET @@session.binlog_format = STATEMENT',
then subsequent DML statements will be written in row format despite
binlog_format=STATEMENT. Because the binlog format can't be reset to
statement based by 'reset_current_stmt_binlog_row_based' function.
If binlog_format=ROW, there are open temporary tables, and an unsafe statement
is executed, then the statement 'SET @@session.binlog_format = MIXED' generates
the error:
"Cannot switch out of the row-based binary log format when the session has open
temporary tables"
However, it is safe to switch to MIXED mode because events in row format are allowed.
To fix the above two problems, generate ER_TEMP_TABLE_PREVENTS_SWITCH_OUT_OF_RBR
and forbid switching from MIXED or ROW to STATEMENT when there are open temp
tables and we are logging in row format. There is no error in any other case.
In RBR, DDL statement will change binlog format to non row-based
format before it is binlogged, but the binlog format was not be
restored, and then manipulating a temporary table can not reset binlog
format to row-based format rightly. So that the manipulated statement
is binlogged with statement-based format.
To fix the problem, restore the state of binlog format after the DDL
statement is binlogged.
