BUG#46364 introduced the flag binlog_direct_non_transactional_updates which
would make N-changes to be written to the binary log upon committing the
statement when "ON". On the other hand, when "OFF" the option was supposed
to mimic the behavior in 5.1. However, the implementation was not mimicking
the behavior correctly and the following bugs popped up:
Case #1: N-changes executed within a transaction would go into
the S-cache. When later in the same transaction a
T-change occurs, N-changes following it were written
to the T-cache instead of the S-cache. In some cases,
this raises problems. For example, a
Table_map_log_event being written initially into the
S-cache, together with the initial N-changes, would be
absent from the T-cache. This would log N-changes
orphaned from a Table_map_log_event (thence discarded
at the slave). (MIXED and ROW)
Case #2: When rolling back a transaction, the N-changes that
might be in the T-cache were disregarded and
truncated along with the T-changes. (MIXED and ROW)
Case #3: When a MIXED statement (TN) is ahead of any other
T-changes in the transaction and it fails, it is kept
in the T-cache until the transaction ends. This is
not the case in 5.1 or Betony (5.5.2). In these, the
failed TN statement would be written to the binlog at
the same instant it had failed and not deferred until
transaction end. (SBR)
To fix these problems, we have decided to do what follows:
For Case #1 and #2, we circumvent them:
1. by not letting binlog_direct_non_transactional_updates
affect MIXED and RBR. These modes will keep the behavior
provided by WL#2687. Although this will make Celosia to
behave differently from 5.1, an execution will be always
safe under such modes in the sense that slaves will never
go out sync. In 5.1, using either MIXED or ROW while
mixing N-statements and T-statements was not safe.
For Case #3, we don't actually fix it. We:
1. keep it and make all MIXED statements whether they end
up failing or not or whether they are up front in the
transaction or after some transactional change to always
be stored in the T-cache. This means that it is written
to the binary log on transaction commit/rollback only.
2. We make the warning message even more specific about the
MIXED statement and SBR.
mysql-test/extra/rpl_tests/rpl_mixing_engines.test:
Updated the test case to avoid checking inconsistencies between the master and slave
when session.binlog_direct_non_transactional_updates is ON and the format is statement.
In this scenario, they will diverge because a counter (within a triger) is incremented
and associated to the issued statement. However, an n-statement is logged ahead of
the transaction and thus is not executed by the same order in the slave and thus gets
a different value from the counter.
mysql-test/suite/binlog/r/binlog_multi_engine.result:
Updated the test case with the new error ER_BINLOG_UNSAFE_MIXED_STATEMENT.
mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result:
Updated the test case with the new error ER_BINLOG_UNSAFE_MIXED_STATEMENT.
mysql-test/suite/ndb/r/ndb_binlog_format.result:
Updated the test case with the new error ER_BINLOG_UNSAFE_MIXED_STATEMENT.
mysql-test/suite/rpl/r/rpl_concurrency_error.result:
Updated the test case with the new error ER_BINLOG_UNSAFE_MIXED_STATEMENT.
mysql-test/suite/rpl/r/rpl_stm_binlog_max_cache_size.result:
Updated the test case with the new error ER_BINLOG_UNSAFE_MIXED_STATEMENT.
mysql-test/suite/rpl/r/rpl_stm_mixing_engines.result:
Updated the test case with the new error ER_BINLOG_UNSAFE_MIXED_STATEMENT.
mysql-test/suite/rpl/r/rpl_stm_stop_middle_group.result:
Updated the test case with the new error ER_BINLOG_UNSAFE_MIXED_STATEMENT.
sql/log.cc:
Checked if either a trx-cache or a non-trx-cache should be used.
If bin_log_direct_non_trans_update is active or the format is either
MIXED or ROW, the cache to be used depends on the flag is_transactional.
When the format is STMT, the non-trx-cache should be used if the statement
is non-transactional and the trx-cache is empty, i.e. if any transactional
statement has not committed yet. Otherwise, the trx-cache should be used.
sql/share/errmsg-utf8.txt:
Added the new unsafe error ER_BINLOG_UNSAFE_MIXED_STATEMENT.
sql/sql_class.cc:
Started printing ER_BINLOG_UNSAFE_MIXED_STATEMENT, when there
is a mixed-statement.
Organized the names of the variables and added comments.
sql/sql_lex.cc:
Added the new unsafe error ER_BINLOG_UNSAFE_MIXED_STATEMENT.
sql/sql_lex.h:
Added the new unsafe error ER_BINLOG_UNSAFE_MIXED_STATEMENT.
mysql-test/t/disabled.def:
Restore disabled ssl tests: SSL certificates were updated.
Disable sp_sync.test, the test case can't work in next-4284.
mysql-test/t/partition_innodb.test:
Disable parsing of the test case for Bug#47343,
the test can not work in next-4284.
mysql-test/t/ps_ddl.test:
Update results (CREATE TABLE IF NOT EXISTS takes
into account existence of the temporary table).
Cherry-pick a fix Bug#37148 from next-mr, to preserve
file ids of the added files, and ensure that all the necessary
changes have been pulled.
Since initially Bug#37148 was null-merged into 6.0,
the changeset that is now being cherry-picked was likewise
null merged into next-4284.
Now that Bug#37148 has been reapplied to 6.0, try to make
it work with next-4284. This is also necessary to be able
to pull other changes from 5.1-rep into next-4284.
To resolve the merge issues use this changeset applied
to 6.0:
revid:jperkin@sun.com-20091216103628-ylhqf7s6yegui2t9
revno: 3776.1.1
committer: He Zhenxing <zhenxing.he@sun.com>
branch nick: 6.0-codebase-bugfixing
timestamp: Thu 2009-12-17 17:02:50 +0800
message:
Fix merge problem with Bug#37148
Conflicts:
- mysql-test/r/mysqld--help-win.result
- sql/sys_vars.cc
Original revsion (in next-mr-bugfixing):
------------------------------------------------------------
revno: 2971 [merge]
revision-id: alfranio.correia@sun.com-20100121210527-rbuheu5rnsmcakh1
committer: Alfranio Correia <alfranio.correia@sun.com>
branch nick: mysql-next-mr-bugfixing
timestamp: Thu 2010-01-21 21:05:27 +0000
message:
BUG#46364 MyISAM transbuffer problems (NTM problem)
It is well-known that due to concurrency issues, a slave can become
inconsistent when a transaction contains updates to both transaction and
non-transactional tables.
In a nutshell, the current code-base tries to preserve causality among the
statements by writing non-transactional statements to the txn-cache which
is flushed upon commit. However, modifications done to non-transactional
tables on behalf of a transaction become immediately visible to other
connections but may not immediately get into the binary log and therefore
consistency may be broken.
In general, it is impossible to automatically detect causality/dependency
among statements by just analyzing the statements sent to the server. This
happen because dependency may be hidden in the application code and it is
necessary to know a priori all the statements processed in the context of
a transaction such as in a procedure. Moreover, even for the few cases that
we could automatically address in the server, the computation effort
required could make the approach infeasible.
So, in this patch we introduce the option
- "--binlog-direct-non-transactional-updates" that can be used to bypass
the current behavior in order to write directly to binary log statements
that change non-transactional tables.
Besides, it is used to enable the WL#2687 which is disabled by default.
------------------------------------------------------------
revno: 2970.1.1
revision-id: alfranio.correia@sun.com-20100121131034-183r4qdyld7an5a0
parent: alik@sun.com-20100121083914-r9rz2myto3tkdya0
committer: Alfranio Correia <alfranio.correia@sun.com>
branch nick: mysql-next-mr-bugfixing
timestamp: Thu 2010-01-21 13:10:34 +0000
message:
BUG#46364 MyISAM transbuffer problems (NTM problem)
It is well-known that due to concurrency issues, a slave can become
inconsistent when a transaction contains updates to both transaction and
non-transactional tables.
In a nutshell, the current code-base tries to preserve causality among the
statements by writing non-transactional statements to the txn-cache which
is flushed upon commit. However, modifications done to non-transactional
tables on behalf of a transaction become immediately visible to other
connections but may not immediately get into the binary log and therefore
consistency may be broken.
In general, it is impossible to automatically detect causality/dependency
among statements by just analyzing the statements sent to the server. This
happen because dependency may be hidden in the application code and it is
necessary to know a priori all the statements processed in the context of
a transaction such as in a procedure. Moreover, even for the few cases that
we could automatically address in the server, the computation effort
required could make the approach infeasible.
So, in this patch we introduce the option
- "--binlog-direct-non-transactional-updates" that can be used to bypass
the current behavior in order to write directly to binary log statements
that change non-transactional tables.
Besides, it is used to enable the WL#2687 which is disabled by default.
condition variable per context instead of one mutex and one conditional
variable for the whole subsystem.
This should increase concurrency in this subsystem.
It also opens the way for further changes which are necessary to solve
such bugs as bug #46272 "MySQL 5.4.4, new MDL: unnecessary deadlock"
and bug #37346 "innodb does not detect deadlock between update and alter
table".
Two other notable changes done by this patch:
- MDL subsystem no longer implicitly acquires global intention exclusive
metadata lock when per-object metadata lock is acquired. Now this has
to be done by explicit calls outside of MDL subsystem.
- Instead of using separate MDL_context for opening system tables/tables
for purposes of I_S we now create MDL savepoint in the main context
before opening tables and rollback to this savepoint after closing
them. This means that it is now possible to get ER_LOCK_DEADLOCK error
even not inside a transaction. This might happen in unlikely case when
one runs DDL on one of system tables while also running DDL on some
other tables. Cases when this ER_LOCK_DEADLOCK error is not justified
will be addressed by advanced deadlock detector for MDL subsystem which
we plan to implement.
mysql-test/include/handler.inc:
Adjusted handler_myisam.test and handler_innodb.test to the fact that
exclusive metadata locks on tables are now acquired according to
alphabetical order of fully qualified table names instead of order
in which tables are mentioned in statement.
mysql-test/r/handler_innodb.result:
Adjusted handler_myisam.test and handler_innodb.test to the fact that
exclusive metadata locks on tables are now acquired according to
alphabetical order of fully qualified table names instead of order
in which tables are mentioned in statement.
mysql-test/r/handler_myisam.result:
Adjusted handler_myisam.test and handler_innodb.test to the fact that
exclusive metadata locks on tables are now acquired according to
alphabetical order of fully qualified table names instead of order
in which tables are mentioned in statement.
mysql-test/r/mdl_sync.result:
Adjusted mdl_sync.test to the fact that exclusive metadata locks on
tables are now acquired according to alphabetical order of fully
qualified table names instead of order in which tables are mentioned
in statement.
mysql-test/t/mdl_sync.test:
Adjusted mdl_sync.test to the fact that exclusive metadata locks on
tables are now acquired according to alphabetical order of fully
qualified table names instead of order in which tables are mentioned
in statement.
sql/events.cc:
Instead of using separate MDL_context for opening system tables we now
create MDL savepoint in the main context before opening such tables
and rollback to this savepoint after closing them. To support this
change methods of THD responsible for saving/restoring open table
state were changed to use Open_tables_backup class which in addition
to Open_table_state has a member for this savepoint. As result code
opening/closing system tables was changed to use Open_tables_backup
instead of Open_table_state class as well.
sql/ha_ndbcluster.cc:
Since manipulations with open table state no longer install proxy
MDL_context it does not make sense to perform them in order to
satisfy assert in mysql_rm_tables_part2(). Removed them per agreement
with Cluster team. This has not broken test suite since scenario in
which deadlock can occur and assertion fails is not covered by tests.
sql/lock.cc:
MDL subsystem no longer implicitly acquires global intention exclusive
metadata lock when per-object exclusive metadata lock is acquired.
Now this has to be done by explicit calls outside of MDL subsystem.
sql/log.cc:
Instead of using separate MDL_context for opening system tables we now
create MDL savepoint in the main context before opening such tables
and rollback to this savepoint after closing them. To support this
change methods of THD responsible for saving/restoring open table
state were changed to use Open_tables_backup class which in addition
to Open_table_state has a member for this savepoint. As result code
opening/closing system tables was changed to use Open_tables_backup
instead of Open_table_state class as well.
sql/mdl.cc:
Changed metadata locking subsystem to use mutex per lock and condition
variable per context instead of one mutex and one conditional variable
for the whole subsystem.
Changed approach to handling of global metadata locks. Instead of
implicitly acquiring intention exclusive locks when user requests
per-object upgradeable or exclusive locks now we require them to be
acquired explicitly in the same way as ordinary metadata locks.
In fact global lock are now ordinary metadata locks in new GLOBAL
namespace.
To implement these changes:
- Removed LOCK_mdl mutex and COND_mdl condition variable.
- Introduced MDL_lock::m_mutex mutexes which protect individual lock
objects.
- Replaced mdl_locks hash with MDL_map class, which has hash for
MDL_lock objects as a member and separate mutex which protects this
hash. Methods of this class allow to find(), find_or_create() or
remove() MDL_lock objects in concurrency-friendly fashion (i.e.
for most common operation, find_or_create(), we don't acquire
MDL_lock::m_mutex while holding MDL_map::m_mutex. Thanks to MikaelR
for this idea and benchmarks!). Added three auxiliary members to
MDL_lock class (m_is_destroyed, m_ref_usage, m_ref_release) to
support this concurrency-friendly behavior.
- Introduced MDL_context::m_ctx_wakeup_cond condition variable to be
used for waiting until this context's pending request can be
satisfied or its thread has to perform actions to resolve potential
deadlock. Context which want to wait add ticket corresponding to the
request to an appropriate queue of waiters in MDL_lock object so
they can be noticed when other contexts change state of lock and be
awaken by them by signalling on MDL_context::m_ctx_wakeup_cond.
As consequence MDL_ticket objects has to be used for any waiting
in metadata locking subsystem including one which happens in
MDL_context::wait_for_locks() method.
Another consequence is that MDL_context is no longer copyable and
can't be saved/restored when working with system tables.
- Made MDL_lock an abstract class, which delegates specifying exact
compatibility matrix to its descendants. Added MDL_global_lock child
class for global lock (The old is_lock_type_compatible() method
became can_grant_lock() method of this class). Added MDL_object_lock
class to represent per-object lock (The old MDL_lock::can_grant_lock()
became its method). Choice between two classes happens based on MDL
namespace in MDL_lock::create() method.
- Got rid of MDL_lock::type member as its meaning became ambigous for
global locks.
- To simplify waking up of contexts waiting for lock split waiting queue
in MDL_lock class in two queues. One for pending requests for exclusive
(including intention exclusive) locks and another for requests for
shared locks.
- Added virtual wake_up_waiters() method to MDL_lock, MDL_global_lock and
MDL_object_lock classes which allows to wake up waiting contexts after
state of lock changes. Replaced old duplicated code with calls to this
method.
- Adjusted MDL_context::try_acquire_shared_lock()/exclusive_lock()/
global_shared_lock(), MDL_ticket::upgrade_shared_lock_to_exclusive_lock()
and MDL_context::release_ticket() methods to use MDL_map and
MDL_lock::m_mutex instead of single LOCK_mdl mutex and wake up
waiters according to the approach described above. The latter method
also was renamed to MDL_context::release_lock().
- Changed MDL_context::try_acquire_shared_lock()/exclusive_lock() and
release_lock() not to handle global locks. They are now supposed to
be taken explicitly like ordinary metadata locks.
- Added helper MDL_context::try_acquire_global_intention_exclusive_lock()
and acquire_global_intention_exclusive_lock() methods.
- Moved common code from MDL_context::acquire_global_shared_lock() and
acquire_global_intention_exclusive_lock() to new method -
MDL_context::acquire_lock_impl().
- Moved common code from MDL_context::try_acquire_shared_lock(),
try_acquire_global_intention_exclusive_lock()/exclusive_lock()
to MDL_context::try_acquire_lock_impl().
- Since acquiring of several exclusive locks can no longer happen under
single LOCK_mdl mutex the approach to it had to be changed. Now we do
it in one by one fashion. This is done in alphabetical order to avoid
deadlocks. Changed MDL_context::acquire_exclusive_locks() accordingly
(as part of this change moved code responsible for acquiring single
exclusive lock to new MDL_context::acquire_exclusive_lock_impl()
method).
- Since we no longer have single LOCK_mdl mutex which protects all
MDL_context::m_is_waiting_in_mdl members using these members to
determine if we have really awaken context holding conflicting
shared lock became inconvinient. Got rid of this member and changed
notify_shared_lock() helper function and process of acquiring
of/upgrading to exclusive lock not to rely on such information.
Now in MDL_context::acquire_exclusive_lock_impl() and
MDL_ticket::upgrade_shared_lock_to_exclusive_lock() we simply
re-try to wake up threads holding conflicting shared locks after
small time out.
- Adjusted MDL_context::can_wait_lead_to_deadlock() and
MDL_ticket::has_pending_conflicting_lock() to use per-lock
mutexes instead of LOCK_mdl. To do this introduced
MDL_lock::has_pending_exclusive_lock() method.
sql/mdl.h:
Changed metadata locking subsystem to use mutex per lock and condition
variable per context instead of one mutex and one conditional variable
for the whole subsystem. In order to implement this change:
- Added MDL_key::cmp() method to be able to sort MDL_key objects
alphabetically. Changed length fields in MDL_key class to uint16
as 16-bit is enough for length of any key.
- Changed MDL_ticket::get_ctx() to return pointer to non-const
object in order to be able to use MDL_context::awake() method
for such contexts.
- Got rid of unlocked versions of can_wait_lead_to_deadlock()/
has_pending_conflicting_lock() methods in MDL_context and
MDL_ticket. We no longer has single mutex which protects all
locks. Thus one always has to use versions of these methods
which acquire per-lock mutexes.
- MDL_request_list type of list now counts its elements.
- Added MDL_context::m_ctx_wakeup_cond condition variable to be used
for waiting until this context's pending request can be satisfied
or its thread has to perform actions to resolve potential deadlock.
Added awake() method to wake up context from such wait.
Addition of condition variable made MDL_context uncopyable.
As result we no longer can save/restore MDL_context when working
with system tables. Instead we create MDL savepoint before opening
those tables and rollback to it once they are closed.
- MDL_context::release_ticket() became release_lock() method.
- Added auxiliary MDL_context::acquire_exclusive_lock_impl() method
which does all necessary work to acquire exclusive lock on one object
but should not be used directly as it does not enforce any asserts
ensuring that no deadlocks are possible.
- Since we no longer need to know if thread trying to acquire exclusive
lock managed to wake up any threads having conflicting shared locks
(as, anyway, we will try to wake up such threads again shortly)
- MDL_context::m_is_waiting_in_mdl member became unnecessary and
notify_shared_lock() no longer needs to be friend of MDL_context.
Changed approach to handling of global metadata locks. Instead of
implicitly acquiring intention exclusive locks when user requests
per-object upgradeable or exclusive locks now we require them to be
acquired explicitly in the same way as ordinary metadata locks.
- Added new GLOBAL namespace for such locks.
- Added new type of lock to be requested MDL_INTENTION_EXCLISIVE.
- Added MDL_context::try_acquire_global_intention_exclusive_lock()
and acquire_global_intention_exclusive_lock() methods.
- Moved common code from MDL_context::acquire_global_shared_lock()
and acquire_global_intention_exclusive_lock() to new method -
MDL_context::acquire_lock_impl().
- Moved common code from MDL_context::try_acquire_shared_lock(),
try_acquire_global_intention_exclusive_lock()/exclusive_lock()
to MDL_context::try_acquire_lock_impl().
- Added helper MDL_context::is_global_lock_owner() method to be
able easily to find what kind of global lock this context holds.
- MDL_context::m_has_global_shared_lock became unnecessary as
global read lock is now represented by ordinary ticket.
- Removed assert in MDL_context::set_lt_or_ha_sentinel() which became
false for cases when we execute LOCK TABLES under global read lock
mode.
sql/mysql_priv.h:
Instead of using separate MDL_context for opening system tables we now
create MDL savepoint in the main context before opening such tables
and rollback to this savepoint after closing them. To support this
change methods of THD responsible for saving/restoring open table
state were changed to use Open_tables_backup class which in addition
to Open_table_state has a member for this savepoint. As result calls
opening/closing system tables were changed to use Open_tables_backup
instead of Open_table_state class as well.
sql/sp.cc:
Instead of using separate MDL_context for opening system tables we now
create MDL savepoint in the main context before opening such tables
and rollback to this savepoint after closing them. To support this
change methods of THD responsible for saving/restoring open table
state were changed to use Open_tables_backup class which in addition
to Open_table_state has a member for this savepoint. As result code
opening/closing system tables was changed to use Open_tables_backup
instead of Open_table_state class as well.
sql/sp.h:
Instead of using separate MDL_context for opening system tables we now
create MDL savepoint in the main context before opening such tables
and rollback to this savepoint after closing them. To support this
change methods of THD responsible for saving/restoring open table
state were changed to use Open_tables_backup class which in addition
to Open_table_state has a member for this savepoint. As result code
opening/closing system tables was changed to use Open_tables_backup
instead of Open_table_state class as well.
sql/sql_base.cc:
close_thread_tables():
Since we no longer use separate MDL_context for opening system
tables we need to avoid releasing all transaction locks when
closing system table. Releasing metadata lock on system table
is now responsibility of THD::restore_backup_open_tables_state().
open_table_get_mdl_lock(),
Open_table_context::recover_from_failed_open():
MDL subsystem no longer implicitly acquires global intention exclusive
metadata lock when per-object upgradable or exclusive metadata lock is
acquired. So this have to be done explicitly from these calls.
Changed Open_table_context class to store MDL_request object for
global intention exclusive lock acquired when opening tables.
open_table():
Do not release metadata lock if we have failed to open table as
this lock might have been acquired by one of previous statements
in transaction, and therefore should not be released.
open_system_tables_for_read()/close_system_tables()/
open_performance_schema_table():
Instead of using separate MDL_context for opening system tables we now
create MDL savepoint in the main context before opening such tables
and rollback to this savepoint after closing them. To support this
change methods of THD responsible for saving/restoring open table
state were changed to use Open_tables_backup class which in addition
to Open_table_state has a member for this savepoint. As result code
opening/closing system tables was changed to use Open_tables_backup
instead of Open_table_state class as well.
close_performance_schema_table():
Got rid of duplicated code.
sql/sql_class.cc:
Instead of using separate MDL_context for opening system tables we now
create MDL savepoint in the main context before opening such tables
and rollback to this savepoint after closing them. To support this
change methods of THD responsible for saving/restoring open table
state were changed to use Open_tables_backup class which in addition
to Open_table_state has a member for this savepoint. Also releasing
metadata lock on system table is now responsibility of
THD::restore_backup_open_tables_state().
Adjusted assert in THD::cleanup() to take into account fact that now
we also use MDL sentinel for global read lock.
sql/sql_class.h:
Instead of using separate MDL_context for opening system tables we now
create MDL savepoint in the main context before opening such tables
and rollback to this savepoint after closing them. As result:
- 'mdl_context' member was moved out of Open_tables_state to THD class.
enter_locked_tables_mode()/leave_locked_tables_mode() had to follow.
- Methods of THD responsible for saving/restoring open table state were
changed to use Open_tables_backup class which in addition to
Open_table_state has a member for this savepoint.
Changed Open_table_context class to store MDL_request object for
global intention exclusive lock acquired when opening tables.
sql/sql_delete.cc:
MDL subsystem no longer implicitly acquires global intention exclusive
metadata lock when per-object exclusive metadata lock is acquired.
Now this has to be done by explicit calls outside of MDL subsystem.
sql/sql_help.cc:
Instead of using separate MDL_context for opening system tables we now
create MDL savepoint in the main context before opening such tables
and rollback to this savepoint after closing them. To support this
change methods of THD responsible for saving/restoring open table
state were changed to use Open_tables_backup class which in addition
to Open_table_state has a member for this savepoint. As result code
opening/closing system tables was changed to use Open_tables_backup
instead of Open_table_state class as well.
sql/sql_parse.cc:
Adjusted assert reload_acl_and_cache() to the fact that global read
lock now takes full-blown metadata lock.
sql/sql_plist.h:
Added support for element counting to I_P_List list template.
One can use policy classes to specify if such counting is needed
or not needed for particular list.
sql/sql_show.cc:
Instead of using separate MDL_context for opening tables for I_S
purposes we now create MDL savepoint in the main context before
opening tables and rollback to this savepoint after closing them.
To support this and similar change for system tables methods of
THD responsible for saving/restoring open table state were changed
to use Open_tables_backup class which in addition to Open_table_state
has a member for this savepoint. As result code opening/closing tables
for I_S purposes was changed to use Open_tables_backup instead of
Open_table_state class as well.
sql/sql_table.cc:
mysql_rm_tables_part2():
Since now global intention exclusive metadata lock is ordinary
metadata lock we no longer can rely that by releasing MDL locks
on all tables we will release all locks acquired by this routine.
So in non-LOCK-TABLES mode we have to release all locks acquired
explicitly.
prepare_for_repair(), mysql_alter_table():
MDL subsystem no longer implicitly acquires global intention
exclusive metadata lock when per-object exclusive metadata lock
is acquired. Now this has to be done by explicit calls outside of
MDL subsystem.
sql/tztime.cc:
Instead of using separate MDL_context for opening system tables we now
create MDL savepoint in the main context before opening such tables
and rollback to this savepoint after closing them. To support this
change methods of THD responsible for saving/restoring open table
state were changed to use Open_tables_backup class which in addition
to Open_table_state has a member for this savepoint. As result code
opening/closing system tables was changed to use Open_tables_backup
instead of Open_table_state class as well.
Also changed code not to use special mechanism for open system tables
when it is not really necessary.
It is well-known that due to concurrency issues, a slave can become
inconsistent when a transaction contains updates to both transaction and
non-transactional tables in statement and mixed modes.
In a nutshell, the current code-base tries to preserve causality among the
statements by writing non-transactional statements to the txn-cache which
is flushed upon commit. However, modifications done to non-transactional
tables on behalf of a transaction become immediately visible to other
connections but may not immediately get into the binary log and therefore
consistency may be broken.
In general, it is impossible to automatically detect causality/dependency
among statements by just analyzing the statements sent to the server. This
happen because dependency may be hidden in the application code and it is
necessary to know a priori all the statements processed in the context of
a transaction such as in a procedure. Moreover, even for the few cases that
we could automatically address in the server, the computation effort
required could make the approach infeasible.
So, in this patch we introduce the option
- "--binlog-direct-non-transactional-updates" that can be used to bypass
the current behavior in order to write directly to binary log statements
that change non-transactional tables.
mysql-test/extra/rpl_tests/rpl_mixing_engines.inc:
Backported this from Celosia to improve the test cases related to the NTM issue.
sql/log.cc:
Checks the --binlog-direct-non-transactional-updates before choosing
to either use the trxn-cache or not.
sql/mysqld.cc:
Introduces the option --binlog-direct-non-transactional-updates.
sql/set_var.cc:
Introduces the option --binlog-direct-non-transactional-updates.
sql/sql_class.h:
Introduces the option --binlog-direct-non-transactional-updates.
variable
The User_var_log_event was not serializing the unsigned
flag. This would cause the slave to always assume signed values.
We fix this by extending the User_var_log_event to also contain
information on the unsigned_flag, meaning that it gets into the
binlog as well, therefore the slave will get this information as
well. Events without information on unsigned flag (old events)
are treated as they were before (always signed: unsigned_flag=
FALSE).
The information on the unsigned_flag, is shipped in an extra byte
appended to the end of the User_var_log_event and added by this
patch. This extra byte holds values for general purpose
User_var_log_event flags which are now packed in the binlog as
well. One of these flags contains information about whether the
value is signed or unsigned (currently this extra byte is only
used to hold data on the unsigned flag, in the future we can use
it to pack extra flags if there is the need to).
Conflicts:
Text conflict in .bzr-mysql/default.conf
Text conflict in mysql-test/extra/rpl_tests/rpl_loaddata.test
Text conflict in mysql-test/r/mysqlbinlog2.result
Text conflict in mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result
Text conflict in mysql-test/suite/binlog/r/binlog_unsafe.result
Text conflict in mysql-test/suite/rpl/r/rpl_insert_id.result
Text conflict in mysql-test/suite/rpl/r/rpl_loaddata.result
Text conflict in mysql-test/suite/rpl/r/rpl_stm_auto_increment_bug33029.result
Text conflict in mysql-test/suite/rpl/r/rpl_udf.result
Text conflict in mysql-test/suite/rpl/t/rpl_slow_query_log.test
Text conflict in sql/field.h
Text conflict in sql/log.cc
Text conflict in sql/log_event.cc
Text conflict in sql/log_event_old.cc
Text conflict in sql/mysql_priv.h
Text conflict in sql/share/errmsg.txt
Text conflict in sql/sp.cc
Text conflict in sql/sql_acl.cc
Text conflict in sql/sql_base.cc
Text conflict in sql/sql_class.h
Text conflict in sql/sql_db.cc
Text conflict in sql/sql_delete.cc
Text conflict in sql/sql_insert.cc
Text conflict in sql/sql_lex.cc
Text conflict in sql/sql_lex.h
Text conflict in sql/sql_load.cc
Text conflict in sql/sql_table.cc
Text conflict in sql/sql_update.cc
Text conflict in sql/sql_view.cc
Conflict adding files to storage/innobase. Created directory.
Conflict because storage/innobase is not versioned, but has versioned children. Versioned directory.
Conflict adding file storage/innobase. Moved existing file to storage/innobase.moved.
Conflict adding files to storage/innobase/handler. Created directory.
Conflict because storage/innobase/handler is not versioned, but has versioned children. Versioned directory.
Contents conflict in storage/innobase/handler/ha_innodb.cc
In auto-commit mode, updating both trx and non-trx tables (i.e. issuing a mixed
statement) causes the following sequence of events:
1 - "Flush trx changes" (MYSQL_BIN_LOG::write) - T1:
1.1 - mutex_lock (&LOCK_log)
1.2 - mutex_lock (&LOCK_prep_xids)
1.3 - increase prepared_xids
1.4 - mutex_unlock (&LOCK_prep_xids)
1.5 - mutex_unlock (&LOCK_log)
2 - "Flush non-trx changes" (MYSQL_BIN_LOG::write) - T1:
2.1 - mutex_lock (&LOCK_log)
2.2 - mutex_unlock (&LOCK_log)
3. "unlog" - T1
3.1 - mutex_lock (&LOCK_prep_xids)
3.2 - decrease prepared xids
3.3 - pthread_cond_signal(&COND_prep_xids);
3.4 - mutex_unlock (&LOCK_prep_xids)
The "FLUSH logs" command produces the following sequence of events:
1 - "FLUSH logs" command (MYSQL_BIN_LOG::new_file_impl) - user thread:
1.1 - mutex_lock (&LOCK_log)
1.2 - mutex_lock (&LOCK_prep_xids)
1.3 - while (prepared_xids) pthread_cond_wait(..., &LOCK_prep_xids);
1.4 - mutex_unlock (&LOCK_prep_xids)
1.5 - mutex_unlock (&LOCK_log)
A deadlock will arise if T1 flushes the trx changes and thus increases
prepared_xids but before it is able to continue the execution and flush the
non-trx changes, an user thread calls the "FLUSH logs" command and wait that
the prepared_xids is decreased and gets to zero. However, T1 cannot proceed
with the call to "Flush non-trx changes" because it will block in the mutex
"LOCK_log" and by consequence cannot complete the execution and call the
unlog to decrease the prepared_xids.
To fix the problem, we ensure that the non-trx changes are always flushed
before the trx changes.
Note that if you call "Flush non-trx changes" and a concurrent "FLUSH logs" is
issued, the "Flush non-trx changes" may block, but a deadlock will never happen
because the prepared_xids will eventually get to zero. Bottom line, there will
not be any transaction able to increase the prepared_xids because they will
block in the mutex "LOCK_log" (MYSQL_BIN_LOG::write) and those that increased
the prepared_xids will eventually commit and decrease the prepared_xids.
Conflicts:
Conflict adding files to server-tools. Created directory.
Conflict because server-tools is not versioned, but has versioned children. Versioned directory.
Conflict adding files to server-tools/instance-manager. Created directory.
Conflict because server-tools/instance-manager is not versioned, but has versioned children. Versioned directory.
Contents conflict in server-tools/instance-manager/instance_map.cc
Contents conflict in server-tools/instance-manager/listener.cc
Contents conflict in server-tools/instance-manager/options.cc
Contents conflict in server-tools/instance-manager/user_map.cc
Bug#16565 mysqld --help --verbose does not order variablesBug#20413 sql_slave_skip_counter is not shown in show variables
Bug#20415 Output of mysqld --help --verbose is incomplete
Bug#25430 variable not found in SELECT @@global.ft_max_word_len;
Bug#32902 plugin variables don't know their names
Bug#34599 MySQLD Option and Variable Reference need to be consistent in formatting!
Bug#34829 No default value for variable and setting default does not raise error
Bug#34834 ? Is accepted as a valid sql mode
Bug#34878 Few variables have default value according to documentation but error occurs
Bug#34883 ft_boolean_syntax cant be assigned from user variable to global var.
Bug#37187 `INFORMATION_SCHEMA`.`GLOBAL_VARIABLES`: inconsistent status
Bug#40988 log_output_basic.test succeeded though syntactically false.
Bug#41010 enum-style command-line options are not honoured (maria.maria-recover fails)
Bug#42103 Setting key_buffer_size to a negative value may lead to very large allocations
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
Bug#44797 plugins w/o command-line options have no disabling option in --help
Bug#46314 string system variables don't support expressions
Bug#46470 sys_vars.max_binlog_cache_size_basic_32 is broken
Bug#46586 When using the plugin interface the type "set" for options caused a crash.
Bug#47212 Crash in DBUG_PRINT in mysqltest.cc when trying to print octal number
Bug#48758 mysqltest crashes on sys_vars.collation_server_basic in gcov builds
Bug#49417 some complaints about mysqld --help --verbose output
Bug#49540 DEFAULT value of binlog_format isn't the default value
Bug#49640 ambiguous option '--skip-skip-myisam' (double skip prefix)
Bug#49644 init_connect and \0
Bug#49645 init_slave and multi-byte characters
Bug#49646 mysql --show-warnings crashes when server dies
CMakeLists.txt:
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
client/mysql.cc:
don't crash with --show-warnings when mysqld dies
config/ac-macros/plugins.m4:
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
include/my_getopt.h:
comments
include/my_pthread.h:
fix double #define
mysql-test/mysql-test-run.pl:
run sys_vars suite by default
properly recognize envirinment variables (e.g. MTR_MAX_SAVE_CORE) set to 0
escape gdb command line arguments
mysql-test/suite/sys_vars/r/rpl_init_slave_func.result:
init_slave+utf8 bug
mysql-test/suite/sys_vars/t/rpl_init_slave_func.test:
init_slave+utf8 bug
mysys/my_getopt.c:
Bug#34599 MySQLD Option and Variable Reference need to be consistent in formatting!
Bug#46586 When using the plugin interface the type "set" for options caused a crash.
Bug#49640 ambiguous option '--skip-skip-myisam' (double skip prefix)
mysys/typelib.c:
support for flagset
sql/ha_ndbcluster.cc:
backport from telco tree
sql/item_func.cc:
Bug#49644 init_connect and \0
Bug#49645 init_slave and multi-byte characters
sql/sql_builtin.cc.in:
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
sql/sql_plugin.cc:
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
Bug#32902 plugin variables don't know their names
Bug#44797 plugins w/o command-line options have no disabling option in --help
sql/sys_vars.cc:
all server variables are defined here
storage/myisam/ft_parser.c:
remove unnecessary updates of param->quot
storage/myisam/ha_myisam.cc:
myisam_* variables belong here
strings/my_vsnprintf.c:
%o and %llx
unittest/mysys/my_vsnprintf-t.c:
%o and %llx tests
vio/viosocket.c:
bugfix: fix @@wait_timeout to work with socket timeouts (vs. alarm thread)
The problem is a somewhat common misusage of the strmake function.
The strmake(dst, src, len) function writes at most /len/ bytes to
the string pointed to by src, not including the trailing null byte.
Hence, if /len/ is the exact length of the destination buffer, a
one byte buffer overflow can occur if the length of the source
string is equal to or greater than /len/.
client/mysqldump.c:
Make room for the trailing null byte.
libmysql/libmysql.c:
Add comment, there is enough room in the buffer.
Increase buffer length, two strings are concatenated.
libmysqld/lib_sql.cc:
Make room for the trailing null byte.
mysys/default.c:
Make room for the trailing null bytes.
mysys/mf_pack.c:
Make room for the trailing null byte.
server-tools/instance-manager/commands.cc:
Copy only if overflow isn't possible in both cases.
server-tools/instance-manager/listener.cc:
Make room for the trailing null byte.
sql/log.cc:
Make room for the trailing null byte.
sql/sp_pcontext.h:
Cosmetic fix.
sql/sql_acl.cc:
MAX_HOSTNAME already specifies space for the trailing null byte.
sql/sql_parse.cc:
Make room for the trailing null byte.
sql/sql_table.cc:
Make room for the trailing null byte.
Bug #36098 Audit plugin (wl 3771) feature disabled in 6.0
avoid recusrive locking of LOCK_plugin
include/mysql/plugin_audit.h:
fix incorrect version
sql/log.cc:
move the common code to a shared header
sql/mysqld.cc:
restore the deleted functionality
sql/set_var.cc:
remove unused parameter
sql/sql_audit.h:
two inline convenience functions
sql/sql_parse.cc:
use a simplified convenience call
sql/sql_plugin.cc:
unlock LOCK_plugin for plugin->init() call, add missing OOM check,
issue "unknown variable" error in find_sys_var, not down the stack
WL#3771
"Audit Plugin Interface"
Implement new plug-in type - AUDIT
New plug-in: audit_null
simply increments counter for how many times it was called.
include/Makefile.am:
wl3771
add new headers to distribution
include/mysql/plugin.h:
wl3771
define new AUDIT plugin type
Split out fulltext plugin type into its own header
include/mysql/plugin.h.pp:
wl3771
no real API change, just re-arranged some code
include/mysql/plugin_audit.h:
wl3771
pluggable audit interface
include/mysql/plugin_ftparser.h:
wl3771
Split out fulltext plugin type into its own header
libmysqld/CMakeLists.txt:
wl3771
add sql_audit.cc to build
libmysqld/Makefile.am:
wl3771
add sql_audit.cc to build
plugin/audit_null:
wl3771
an example plugin for testing pluggable audit interface
plugin/audit_null/Makefile.am:
wl3771
an example plugin for testing pluggable audit interface
plugin/audit_null/audit_null.c:
wl3771
an example plugin for testing pluggable audit interface
plugin/audit_null/plug.in:
wl3771
an example plugin for testing pluggable audit interface
sql/CMakeLists.txt:
wl3771
add sql_audit.cc to build
sql/Makefile.am:
wl3771
add sql_audit.cc to build
sql/event_queue.cc:
wl3771
release audit resources before waiting
sql/log.cc:
wl3771
add general audit call for log
sql/mysqld.cc:
wl3771
add audit initialize/finalize
add general audit call for error
sql/sql_audit.cc:
wl3771
pluggable audit interface implementation
sql/sql_audit.h:
wl3771
pluggable audit interface implementation
sql/sql_class.cc:
wl3771
add thd audit init/deinit calls
sql/sql_class.h:
wl3771
add required data structures for audit to THD
sql/sql_connect.cc:
wl3771
release audit resources before waiting
sql/sql_insert.cc:
wl3771
release audit plugins before waiting
sql/sql_parse.cc:
wl3771
add general audit call for results
sql/sql_plugin.cc:
wl3771
add declarations for audit plugin type
for InnoDB
The class Field_bit_as_char stores the metadata for the
field incorrecly because bytes_in_rec and bit_len are set
to (field_length + 7 ) / 8 and 0 respectively, while
Field_bit has the correct values field_length / 8 and
field_length % 8.
Solved the problem by re-computing the values for the
metadata based on the field_length instead of using the
bytes_in_rec and bit_len variables.
To handle compatibility with old server, a table map
flag was added to indicate that the bit computation is
exact. If the flag is clear, the slave computes the
number of bytes required to store the bit field and
compares that instead, effectively allowing replication
*without conversion* from any field length that require
the same number of bytes to store.
mysql-test/suite/rpl/t/rpl_typeconv_innodb.test:
Adding test to check compatibility for bit field
replication when using InnoDB.
sql/field.cc:
Extending compatible_field_size() with flags from
table map to allow fields to check master info.
sql/field.h:
Extending compatible_field_size() with flags from
table map to allow fields to check master info.
sql/log.cc:
Removing table map flags since they are not used
outside table map class.
sql/log_event.cc:
Removing flags parameter from table map constructor
since it is not used and does not have to be exposed.
sql/log_event.h:
Adding flag to denote that bit length for bit field type
is exact and not potentially rounded to even bytes.
sql/rpl_utility.cc:
Adding fields to table_def to store table map flags.
sql/rpl_utility.h:
Removing obsolete comment and adding flags to store
table map flags from master.
Problem is that purge_logs implementation in ndb (ndbcluster_binlog_index_purge_file)
calls mysql_parse (with (thd->options & OPTION_BIN_LOG) === 0))
but MYSQL_BIN_LOG first takes LOCK_log and then checks thd->options
Solution in this patch, changes so that rotate_and_purge does not hold
LOCK_log when calling purge_logs_before_date. I think this is safe
as other "purge"-function(s) is called wo/ holding LOCK_log, e.g purge_master_logs
