mirror/mariadb
1
0
Fork 0
mirror of https://github.com/MariaDB/server.git synced 2025-01-17 20:42:30 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions
66410 commits 2784 branches 343 tags 1.6 GiB
Commit graph

508 commits

Author SHA1 Message Date
Konstantin Osipov
29c4873ad5 Commit on behalf of Dmitry Lenev. 
Merge his patch for Bug#52044 into 5.5, and apply 
review comments.
 2010-08-12 17:50:23 +04:00
unknown
cb63e6d021 Manual merge 2010-07-29 11:24:35 +08:00
unknown
2124538d9c BUG#49124 Security issue with /*!-versioned */ SQL statements on Slave 
/*![:version:] Query Code */, where [:version:] is a sequence of 5 
digits representing the mysql server version(e.g /*!50200 ... */),
is a special comment that the query in it can be executed on those 
servers whose versions are larger than the version appearing in the 
comment. It leads to a security issue when slave's version is larger 
than master's. A malicious user can improve his privileges on slaves. 
Because slave SQL thread is running with SUPER privileges, so it can
execute queries that he/she does not have privileges on master.

This bug is fixed with the logic below: 
- To replace '!' with ' ' in the magic comments which are not applied on
  master. So they become common comments and will not be applied on slave.

- Example:
  'INSERT INTO t1 VALUES (1) /*!10000, (2)*/ /*!99999 ,(3)*/
  will be binlogged as
  'INSERT INTO t1 VALUES (1) /*!10000, (2)*/ /* 99999 ,(3)*/

mysql-test/suite/rpl/t/rpl_conditional_comments.test:
  Test the patch for this bug.
sql/mysql_priv.h:
  Rename inBuf as rawBuf and remove the const limitation.
sql/sql_lex.cc:
  To replace '!' with ' ' in the magic comments which are not applied on
  master.
sql/sql_lex.h:
  Remove the const limitation on parameter buff, as it can be modified in the function since
  this patch.
  Add member function yyUnput for Lex_input_stream. It set a character back the query buff.
sql/sql_parse.cc:
  Rename inBuf as rawBuf and remove the const limitation.
sql/sql_partition.cc:
  Remove the const limitation on parameter part_buff, as it can be modified in the function since
  this patch.
sql/sql_partition.h:
  Remove the const limitation on parameter part_buff, as it can be modified in the function since
  this patch.
sql/table.h:
  Remove the const limitation on variable partition_info, as it can be modified since
  this patch.
 2010-07-29 11:00:57 +08:00
Dmitry Lenev
00496b7acd Fix for bug #52044 "FLUSH TABLES WITH READ LOCK and FLUSH 
TABLES <list> WITH READ LOCK are incompatible".

The problem was that FLUSH TABLES <list> WITH READ LOCK
which was issued when other connection has acquired global
read lock using FLUSH TABLES WITH READ LOCK was blocked
and has to wait until global read lock is released.

This issue stemmed from the fact that FLUSH TABLES <list>
WITH READ LOCK implementation has acquired X metadata locks
on tables to be flushed. Since these locks required acquiring
of global IX lock this statement was incompatible with global
read lock.

This patch addresses problem by using SNW metadata type of
lock for tables to be flushed by FLUSH TABLES <list> WITH
READ LOCK. It is OK to acquire them without global IX lock
as long as we won't try to upgrade those locks. Since SNW
locks allow concurrent statements using same table FLUSH
TABLE <list> WITH READ LOCK now has to wait until old
versions of tables to be flushed go away after acquiring
metadata locks. Since such waiting can lead to deadlock
MDL deadlock detector was extended to take into account
waits for flush and resolve such deadlocks.

As a bonus code in open_tables() which was responsible for
waiting old versions of tables to go away was refactored.
Now when we encounter old version of table in open_table()
we don't back-off and wait for all old version to go away,
but instead wait for this particular table to be flushed.
Such approach supported by deadlock detection should reduce
number of scenarios in which FLUSH TABLES aborts concurrent
multi-statement transactions.

Note that active FLUSH TABLES <list> WITH READ LOCK still
blocks concurrent FLUSH TABLES WITH READ LOCK statement
as the former keeps tables open and thus prevents the
latter statement from doing flush.

mysql-test/include/handler.inc:
  Adjusted test case after changing status which is set
  when FLUSH TABLES waits for tables to be flushed from
  "Flushing tables" to "Waiting for table".
mysql-test/r/flush.result:
  Added test which checks that "flush tables <list> with
  read lock" is compatible with active "flush tables with
  read lock" but not vice-versa. This test also covers
  bug #52044 "FLUSH TABLES WITH READ LOCK and FLUSH TABLES
  <list> WITH READ LOCK are incompatible".
mysql-test/r/mdl_sync.result:
  Added scenarios in which wait for table to be flushed
  causes deadlocks to the coverage of MDL deadlock detector.
mysql-test/suite/perfschema/r/dml_setup_instruments.result:
  Adjusted test results after removal of COND_refresh
  condition variable.
mysql-test/suite/perfschema/r/server_init.result:
  Adjusted test and its results after removal of COND_refresh
  condition variable.
mysql-test/suite/perfschema/t/server_init.test:
  Adjusted test and its results after removal of COND_refresh
  condition variable.
mysql-test/t/flush.test:
  Added test which checks that "flush tables <list> with
  read lock" is compatible with active "flush tables with
  read lock" but not vice-versa. This test also covers
  bug #52044 "FLUSH TABLES WITH READ LOCK and FLUSH TABLES
  <list> WITH READ LOCK are incompatible".
mysql-test/t/kill.test:
  Adjusted test case after changing status which is set
  when FLUSH TABLES waits for tables to be flushed from
  "Flushing tables" to "Waiting for table".
mysql-test/t/lock_multi.test:
  Adjusted test case after changing status which is set
  when FLUSH TABLES waits for tables to be flushed from
  "Flushing tables" to "Waiting for table".
mysql-test/t/mdl_sync.test:
  Added scenarios in which wait for table to be flushed
  causes deadlocks to the coverage of MDL deadlock detector.
sql/ha_ndbcluster.cc:
  Adjusted code after adding one more parameter for
  close_cached_tables() call - timeout for waiting for
  table to be flushed.
sql/ha_ndbcluster_binlog.cc:
  Adjusted code after adding one more parameter for
  close_cached_tables() call - timeout for waiting for
  table to be flushed.
sql/lock.cc:
  Removed COND_refresh condition variable. See comment
  for sql_base.cc for details.
sql/mdl.cc:
  Now MDL deadlock detector takes into account information
  about waits for table flushes when searching for deadlock.
  To implement this change:
  - Declaration of enum_deadlock_weight and
    Deadlock_detection_visitor were moved to mdl.h header
    to make them available to the code in table.cc which
    implements deadlock detector traversal through edges
    of waiters graph representing waiting for flush.
  - Since now MDL_context may wait not only for metadata
    lock but also for table to be flushed an abstract
    Wait_for_edge class was introduced. Its descendants
    MDL_ticket and Flush_ticket incapsulate specifics
    of inspecting waiters graph when following through
    edge representing wait of particular type.
  
  We no longer require global IX metadata lock when acquiring
  SNW or SNRW locks. Such locks are needed only when metadata
  locks of these types are upgraded to X locks. This allows
  to use SNW locks in FLUSH TABLES <list> WITH READ LOCK
  implementation and keep the latter compatible with global
  read lock.
sql/mdl.h:
  Now MDL deadlock detector takes into account information
  about waits for table flushes when searching for deadlock.
  To implement this change:
  - Declaration of enum_deadlock_weight and
    Deadlock_detection_visitor were moved to mdl.h header
    to make them available to the code in table.cc which
    implements deadlock detector traversal through edges
    of waiters graph representing waiting for flush.
  - Since now MDL_context may wait not only for metadata
    lock but also for table to be flushed an abstract
    Wait_for_edge class was introduced. Its descendants
    MDL_ticket and Flush_ticket incapsulate specifics
    of inspecting waiters graph when following through
    edge representing wait of particular type.
  - Deadlock_detection_visitor now has m_table_shares_visited
    member which allows to support recursive locking for
    LOCK_open. This is required when deadlock detector
    inspects waiters graph which contains several edges
    representing waits for flushes or needs to come through
    the such edge more than once.
sql/mysqld.cc:
  Removed COND_refresh condition variable. See comment
  for sql_base.cc for details.
sql/mysqld.h:
  Removed COND_refresh condition variable. See comment
  for sql_base.cc for details.
sql/sql_base.cc:
  Changed approach to how threads are waiting for table
  to be flushed. Now thread that wants to wait for old
  table to go away subscribes for notification by adding
  Flush_ticket to table's share and waits using
  MDL_context::m_wait object. Once table gets flushed
  (i.e. all tables are closed and table share is ready
  to be destroyed) all such waiters are notified
  individually.
  Thanks to this change MDL deadlock detector can take
  such waits into account.
  
  To implement this/as result of this change:
  - tdc_wait_for_old_versions() was replaced with
    tdc_wait_for_old_version() which waits for individual
    old share to go away and which is called by open_table()
    after finding out that share is outdated. We don't
    need to perform back-off before such waiting thanks
    to the fact that deadlock detector now sees such waits.
  - As result Open_table_ctx::m_mdl_requests became
    unnecessary and was removed. We no longer allocate
    copies of MDL_request objects on MEM_ROOT when
    MYSQL_OPEN_FORCE_SHARED/SHARED_HIGH_PRIO flags are
    in effect.
  - close_cached_tables() and tdc_wait_for_old_version()
    share code which implements waiting for share to be
    flushed - the both use TABLE_SHARE::wait_until_flush()
    method. Thanks to this close_cached_tables() supports
    timeouts and has extra parameter for this.
  - Open_table_context::OT_MDL_CONFLICT enum element was
    renamed to OT_CONFLICT as it is now also used in cases
    when back-off is required to resolve deadlock caused
    by waiting for flush and not metadata lock.
  - In cases when we discover that current connection tries
    to open tables from different generation we now simply
    back-off and restart process of opening tables. To
    support this Open_table_context::OT_REOPEN_TABLES enum
    element was added.
  - COND_refresh condition variable became unnecessary and
    was removed.
  - mysql_notify_thread_having_shared_lock() no longer wakes
    up connections waiting for flush as all such connections
    can be waken up by deadlock detector if necessary.
sql/sql_base.h:
  - close_cached_tables() now has one more parameter -
    timeout for waiting for table to be flushed.
  - Open_table_context::OT_MDL_CONFLICT enum element was
    renamed to OT_CONFLICT as it is now also used in cases
    when back-off is required to resolve deadlock caused
    by waiting for flush and not metadata lock.
    Added new OT_REOPEN_TABLES enum element to be used in
    cases when we need to restart open tables process even
    in the middle of transaction.
  - Open_table_ctx::m_mdl_requests became unnecessary and
    was removed.
sql/sql_class.h:
  Added assert ensuring that we won't use LOCK_open mutex
  with THD::enter_cond(). Otherwise deadlocks can arise in
  MDL deadlock detector.
sql/sql_parse.cc:
  Changed FLUSH TABLES <list> WITH READ LOCK to take SNW
  metadata locks instead of X locks on tables to be flushed.
  Since we no longer require global IX lock to be taken
  when SNW locks are taken this makes this statement
  compatible with FLUSH TABLES WITH READ LOCK statement.
  Since SNW locks allow other connections to have table
  opened FLUSH TABLES <list> WITH READ LOCK now has to
  wait during open_tables() for old version to go away.
  Such waits can lead to deadlocks which will be detected
  by MDL deadlock detector which now takes waits for table
  to be flushed into account.
  
  Also adjusted code after adding one more parameter for
  close_cached_tables() call - timeout for waiting for
  table to be flushed.
sql/sql_yacc.yy:
  FLUSH TABLES <list> WITH READ LOCK now needs only SNW
  metadata locks on tables.
sql/sys_vars.cc:
  Adjusted code after adding one more parameter for
  close_cached_tables() call - timeout for waiting for
  table to be flushed.
sql/table.cc:
  Implemented new approach to how threads are waiting for
  table to be flushed. Now thread that wants to wait for
  old table to go away subscribes for notification by
  adding Flush_ticket to table's share and waits using
  MDL_context::m_wait object. Once table gets flushed
  (i.e. all tables are closed and table share is ready
  to be destroyed) all such waiters are notified
  individually. This change allows to make such waits
  visible inside of MDL deadlock detector.
  To do it:
  
  - Added list of waiters/Flush_tickets to TABLE_SHARE
    class.
  - Changed free_table_share() to postpone freeing of
    share memory until last waiter goes away and to
    wake up subscribed waiters.
  - Added TABLE_SHARE::wait_until_flushed() method which
    implements subscription to the list of waiters for
    table to be flushed and waiting for this event.
  
  Implemented interface which allows to expose waits for
  flushes to MDL deadlock detector:
  
  - Introduced Flush_ticket class a descendant of
    Wait_for_edge class.
  - Added TABLE_SHARE::find_deadlock() method which allows
    deadlock detector to find out what contexts are still
    using old version of table in question (i.e. to find
    out what contexts are waited for by owner of
    Flush_ticket).
sql/table.h:
  In order to support new strategy of waiting for table flush
  (see comment for table.cc for details) added list of
  waiters/Flush_tickets to TABLE_SHARE class.
  
  Implemented interface which allows to expose waits for
  flushes to MDL deadlock detector:
  - Introduced Flush_ticket class a descendant of
    Wait_for_edge class.
  - Added TABLE_SHARE::find_deadlock() method which allows
    deadlock detector to find out what contexts are still
    using old version of table in question (i.e. to find
    out what contexts are waited for by owner of
    Flush_ticket).
 2010-07-27 17:34:58 +04:00
Gleb Shchepa
ef4c0f68d1 Bug #30584: delete with order by and limit clauses does not 
use limit efficiently
Bug #36569: UPDATE ... WHERE ... ORDER BY... always does a
            filesort even if not required

Also two bugs reported after QA review (before the commit
of bugs above to public trees, no documentation needed):

Bug #53737: Performance regressions after applying patch
            for bug 36569
Bug #53742: UPDATEs have no effect after applying patch
            for bug 36569


Execution of single-table UPDATE and DELETE statements did not use the 
same optimizer as was used in the compilation of SELECT statements. 
Instead, it had an optimizer of its own that did not take into account 
that you can omit sorting by retrieving rows using an index.

Extra optimization has been added: when applicable, single-table 
UPDATE/DELETE statements use an existing index instead of filesort. A 
corresponding SELECT query would do the former.

Also handling of the DESC ordering expression has been added when
reverse index scan is applicable.

From now on most single table UPDATE and DELETE statements show the 
same disk access patterns as the corresponding SELECT query. We verify 
this by comparing the result of SHOW STATUS LIKE 'Sort%

Currently the get_index_for_order function 
a) checks quick select index (if any) for compatibility with the
   ORDER expression list or
b) chooses the cheapest available compatible index, but only if 
   the index scan is cheaper than filesort.
Second way is implemented by the new test_if_cheaper_ordering
function (extracted part the test_if_skip_sort_order()).



mysql-test/r/log_state.result:
  Updated result for optimized query, bug #36569.
mysql-test/r/single_delete_update.result:
  Test case for bug #30584, bug #36569 and bug #53742.
mysql-test/r/update.result:
  Updated result for optimized query, bug #30584.
  Note:
  "Handler_read_last 1" omitted, see bug 52312:
  lost Handler_read_last status variable.
mysql-test/t/single_delete_update.test:
  Test case for bug #30584, bug #36569 and bug #53742.
sql/opt_range.cc:
  Bug #30584, bug #36569: UPDATE/DELETE ... WHERE ... ORDER BY...
                          always does a filesort even if not required
  
  * get_index_for_order() has been rewritten entirely and moved
    to sql_select.cc
  
  New QUICK_RANGE_SELECT::make_reverse method has been added.
sql/opt_range.h:
  Bug #30584, bug #36569: UPDATE/DELETE ... WHERE ... ORDER BY...
                          always does a filesort even if not required
  
  * get_index_for_order() has been rewritten entirely and moved
    to sql_select.cc
  
  New functions:
  * QUICK_SELECT_I::make_reverse()
  * SQL_SELECT::set_quick()
sql/records.cc:
  Bug #30584, bug #36569: UPDATE/DELETE ... WHERE ... ORDER BY...
                          always does a filesort even if not required
  
  * init_read_record_idx() has been modified to allow reverse index scan
  
  New functions:
  * rr_index_last()
  * rr_index_desc()
sql/records.h:
  Bug #30584, bug #36569: UPDATE/DELETE ... WHERE ... ORDER BY...
                          always does a filesort even if not required
  
  init_read_record_idx() has been modified to allow reverse index scan
sql/sql_delete.cc:
  Bug #30584, bug #36569: UPDATE/DELETE ... WHERE ... ORDER BY...
                          always does a filesort even if not required
      
  mysql_delete: an optimization has been added to skip
  unnecessary sorting with ORDER BY clause where select
  result ordering is acceptable.
sql/sql_select.cc:
  Bug #30584, bug #36569, bug #53737, bug #53742:
    UPDATE/DELETE ... WHERE ... ORDER BY...  always does a filesort
    even if not required
      
  The const_expression_in_where function has been modified
  to accept both Item and Field pointers.
  
  New functions:
  * get_index_for_order()
  * test_if_cheaper_ordering() has been extracted from
    test_if_skip_sort_order() to share with get_index_for_order()
  * simple_remove_const()
sql/sql_select.h:
  Bug #30584, bug #36569: UPDATE/DELETE ... WHERE ... ORDER BY...
                          always does a filesort even if not required
      
  New functions:
  * test_if_cheaper_ordering()
  * simple_remove_const()
  * get_index_for_order()
sql/sql_update.cc:
  Bug #30584, bug #36569: UPDATE/DELETE ... WHERE ... ORDER BY...
                          always does a filesort even if not required
      
  mysql_update: an optimization has been added to skip
  unnecessary sorting with ORDER BY clause where a select
  result ordering is acceptable.
sql/table.cc:
  Bug #30584, bug #36569: UPDATE/DELETE ... WHERE ... ORDER BY...
                          always does a filesort even if not required
  
  New functions:
  * TABLE::update_const_key_parts()
  * is_simple_order()
sql/table.h:
  Bug #30584, bug #36569: UPDATE/DELETE ... WHERE ... ORDER BY...
                          always does a filesort even if not required
  
  New functions:
  * TABLE::update_const_key_parts()
  * is_simple_order()
 2010-06-23 00:32:29 +04:00
Konstantin Osipov
dc00b5dbb2 Merge. 2010-06-17 17:55:00 +04:00
Konstantin Osipov
cc6dabba37 Merge trunk-bugfixing -> trunk-runtime 2010-06-17 17:31:51 +04:00
Dmitry Lenev
981f3f5b11 Small clean-up and pre-requisite for making 
TABLE_SHARE a class.

Remove unused members in TABLE_SHARE.
 2010-06-16 13:04:30 +04:00
Konstantin Osipov
33c57e2f9e WL#5419 "LOCK_open scalability: make tdc_refresh_version 
an atomic counter"

Split the large LOCK_open section in open_table(). 
Do not call open_table_from_share() under LOCK_open.
Remove thd->version.

This fixes
Bug#50589 "Server hang on a query evaluated using a temporary 
table"
Bug#51557 "LOCK_open and kernel_mutex are not happy together"
Bug#49463 "LOCK_table and innodb are not nice when handler 
instances are created".

This patch has effect on storage engines that rely on
ha_open() PSEA method being called under LOCK_open.
In particular:

1) NDB is broken and left unfixed. NDB relies on LOCK_open
being kept as part of ha_open(), since it uses auto-discovery.
While previously the NDB open code was race-prone, now
it simply fails on asserts.

2) HEAP engine had a race in ha_heap::open() when
a share for the same table could be added twice
to the list of shares, or a dangling reference to a share
stored in HEAP handler. This patch aims to address this
problem by 'pinning' the newly created share in the 
internal HEAP engine share list until at least one
handler instance is created using that share.


include/heap.h:
  Add members to HP_CREATE_INFO.
  Declare heap_release_share().
sql/lock.cc:
  Remove thd->version, use thd->open_tables->s->version instead.
sql/repl_failsafe.cc:
  Remove thd->version.
sql/sql_base.cc:
  - close_thread_table(): move handler cleanup code outside the critical section protected by LOCK_open.
  - remove thd->version
  - split the large critical section in open_table() that
  opens a new table from share and is protected by LOCK_open
  into 2 critical sections, thus reducing the critical path.
  - make check_if_table_exists() acquire LOCK_open internally.
  - use thd->open_tables->s->version instead of thd->refresh_version to make sure that all tables in
  thd->open_tables are in the same refresh series.
sql/sql_base.h:
  Add declaration for check_if_table_exists().
sql/sql_class.cc:
  Remove init_open_tables_state(), it's now equal to
  reset_open_tables_state().
sql/sql_class.h:
  Remove thd->version, THD::init_open_tables_state().
sql/sql_plugin.cc:
  Use table->m_needs_reopen to mark the table as stale
  rather than manipulate with thd->version, which is no more.
sql/sql_udf.cc:
  Use table->m_needs_reopen to mark the table as stale
  rather than manipulate with thd->version, which is no more.
sql/table.h:
  Remove an unused variable.
sql/tztime.cc:
  Use table->m_needs_reopen to mark the table as stale
  rather than manipulate with thd->version, which is no more.
storage/heap/CMakeLists.txt:
  Add heap tests to cmake build files.
storage/heap/ha_heap.cc:
  Fix a race when ha_heap::ha_open() could insert two 
  HP_SHARE objects into the internal share list or store
  a dangling reference to a share in ha_heap instance,
  or wrongly set implicit_emptied.
storage/heap/hp_create.c:
  Optionally pin a newly created share in the list of shares
  by increasing its open_count. This is necessary to 
  make sure that a newly created share doesn't disappear while
  a HP_INFO object is being created to reference it.
storage/heap/hp_open.c:
  When adding a new HP_INFO object to the list of objects
  in the heap share, make sure the open_count is not increased
  twice.
storage/heap/hp_test1.c:
  Adjust the test to new function signatures.
storage/heap/hp_test2.c:
  Adjust the test to new function signatures.
 2010-06-11 19:28:18 +04:00
Davi Arnaut
db2fe44c84 Merge of mysql-5.1-bugteam into mysql-trunk-merge. 2010-06-10 22:30:49 -03:00
Davi Arnaut
0f9ddfa9d8 Bug#42733: Type-punning warnings when compiling MySQL -- 
strict aliasing violations.

One somewhat major source of strict-aliasing violations and
related warnings is the SQL_LIST structure. For example,
consider its member function `link_in_list` which takes
a pointer to pointer of type T (any type) as a pointer to
pointer to unsigned char. Dereferencing this pointer, which
is done to reset the next field, violates strict-aliasing
rules and might cause problems for surrounding code that
uses the next field of the object being added to the list.

The solution is to use templates to parametrize the SQL_LIST
structure in order to deference the pointers with compatible
types. As a side bonus, it becomes possible to remove quite
a few casts related to acessing data members of SQL_LIST.

sql/handler.h:
  Use the appropriate template type argument.
sql/item.cc:
  Remove now-unnecessary cast.
sql/item_subselect.cc:
  Remove now-unnecessary casts.
sql/item_sum.cc:
  Use the appropriate template type argument.
  Remove now-unnecessary cast.
sql/mysql_priv.h:
  Move SQL_LIST structure to sql_list.h
  Use the appropriate template type argument.
sql/sp.cc:
  Remove now-unnecessary casts.
sql/sql_delete.cc:
  Use the appropriate template type argument.
  Remove now-unnecessary casts.
sql/sql_derived.cc:
  Remove now-unnecessary casts.
sql/sql_lex.cc:
  Remove now-unnecessary casts.
sql/sql_lex.h:
  SQL_LIST now takes a template type argument which must
  match the type of the elements of the list. Use forward
  declaration when the type is not available, it is used
  in pointers anyway.
sql/sql_list.h:
  Rename SQL_LIST to SQL_I_List. The template parameter is
  the type of object that is stored in the list.
sql/sql_olap.cc:
  Remove now-unnecessary casts.
sql/sql_parse.cc:
  Remove now-unnecessary casts.
sql/sql_prepare.cc:
  Remove now-unnecessary casts.
sql/sql_select.cc:
  Remove now-unnecessary casts.
sql/sql_show.cc:
  Remove now-unnecessary casts.
sql/sql_table.cc:
  Remove now-unnecessary casts.
sql/sql_trigger.cc:
  Remove now-unnecessary casts.
sql/sql_union.cc:
  Remove now-unnecessary casts.
sql/sql_update.cc:
  Remove now-unnecessary casts.
sql/sql_view.cc:
  Remove now-unnecessary casts.
sql/sql_yacc.yy:
  Remove now-unnecessary casts.
storage/myisammrg/ha_myisammrg.cc:
  Remove now-unnecessary casts.
 2010-06-10 17:45:22 -03:00
Konstantin Osipov
90a87cd52d A pre-requisite for WL#5419 "LOCK_open scalability: 
make tdc_refresh_version an 
atomic counter".
Backport the trivial changes from mysql-trunk-iplus.
 2010-06-10 15:43:32 +04:00
Alexander Nozdrin
f29d24b0f8 Manual merge from mysql-trunk-bugfixing. 
Conflicts:
  - BUILD/SETUP.sh
  - mysql-test/mysql-test-run.pl
  - mysql-test/r/partition_error.result
  - mysql-test/t/disabled.def
  - mysql-test/t/partition_error.test
  - sql/share/errmsg-utf8.txt
 2010-06-07 12:47:04 +04:00
Jon Olav Hauglid
142a162c66 manual merge from mysql-trunk-bugfixing 
Conflicts:
Text conflict in mysql-test/r/archive.result
Contents conflict in mysql-test/r/innodb_bug38231.result
Text conflict in mysql-test/r/mdl_sync.result
Text conflict in mysql-test/suite/binlog/t/disabled.def
Text conflict in mysql-test/suite/rpl_ndb/r/rpl_ndb_binlog_format_errors.result
Text conflict in mysql-test/t/archive.test
Contents conflict in mysql-test/t/innodb_bug38231.test
Text conflict in mysql-test/t/mdl_sync.test
Text conflict in sql/sp_head.cc
Text conflict in sql/sql_show.cc
Text conflict in sql/table.cc
Text conflict in sql/table.h
 2010-06-06 13:19:29 +02:00
Alexander Nozdrin
59a9912963 Auto-merge from mysql-trunk. 2010-05-28 09:47:58 +04:00
Davi Arnaut
d108c80ee7 Fix type mismatch. Table names are represented as LEX_STRING 
objects whose length is stored in a size_t type.
 2010-05-27 18:28:24 -03:00
Alexey Kopytov
a2e7f632cf Automerge. 2010-05-27 14:08:44 +04:00
Davi Arnaut
4f18083b08 Bug#42643: InnoDB does not support replication of TRUNCATE TABLE 
The problem was that TRUNCATE TABLE didn't take a exclusive
lock on a table if it resorted to truncating via delete of
all rows in the table. Specifically for InnoDB tables, this
could break proper isolation as InnoDB ends up aborting some
granted locks when truncating a table.

The solution is to take a exclusive metadata lock before
TRUNCATE TABLE can proceed. This guarantees that no other
transaction is using the table.

Incompatible change: Truncate via delete no longer fails
if sql_safe_updates is activated (this was a undocumented
side effect).

libmysqld/CMakeLists.txt:
  Add new files to the build list.
libmysqld/Makefile.am:
  Add new files to the build list.
mysql-test/extra/binlog_tests/binlog_truncate.test:
  Add test case for Bug#42643
mysql-test/include/mix1.inc:
  Update test case as TRUNCATE TABLE now grabs a exclusive lock.
  Ensure that TRUNCATE waits for granted locks on the table.
mysql-test/suite/binlog/t/binlog_truncate_innodb.test:
  As with other data modifying statements, TRUNCATE is still not
  possible in a transaction with isolation level READ COMMITTED
  or READ UNCOMMITED. It would be possible to implement so, but
  it is not worth the effort.
mysql-test/suite/binlog/t/binlog_truncate_myisam.test:
  Test under different binlog formats.
mysql-test/suite/binlog/t/disabled.def:
  Re-enable test case.
mysql-test/t/innodb_bug38231.test:
  Truncate no longer works with row-level locks.
mysql-test/t/mdl_sync.test:
  Ensure that a acquired lock is not given up due to a conflict.
mysql-test/t/partition_innodb_semi_consistent.test:
  End transaction as to release metadata locks.
mysql-test/t/truncate.test:
  A metadata lock is now taken before the object is verified.
sql/CMakeLists.txt:
  Add new files to the build list.
sql/Makefile.am:
  Add new files to the build list.
sql/datadict.cc:
  Introduce a new file specific for data dictionary operations.
sql/datadict.h:
  Add header file.
sql/sql_base.cc:
  Rename data dictionary function.
sql/sql_bitmap.h:
  Include dependency.
sql/sql_delete.cc:
  Move away from relying on mysql_delete() to delete all rows of
  a table. Thus, move any bits related to truncate to sql_truncate.cc
sql/sql_delete.h:
  Remove parameter.
sql/sql_parse.cc:
  Add protection against the global read lock -- a intention
  exclusive lock can be acquired in the truncate path.
sql/sql_show.cc:
  Add sync point for testing scenarios where a pending flush
  is ignored.
sql/sql_truncate.cc:
  Acquire a shared metadata lock before accessing table metadata.
  Upgrade the lock to a exclusive one if the table can be re-created.
  Rework binlog rules to better reflect the requirements.
sql/sql_yacc.yy:
  Set appropriate lock types for table to be truncated.
sql/table.h:
  Move to data dictionary header.
 2010-05-25 17:01:38 -03:00
Alexey Kopytov
1d0acc7754 Bug #53830: !table || (!table->read_set || 
bitmap_is_set(table->read_set, field_index))

UPDATE on an InnoDB table modifying the same index that is used
to satisfy the WHERE condition could trigger a debug assertion
under some circumstances.

Since for engines with the HA_PRIMARY_KEY_IN_READ_INDEX flag
set results of an index scan on a secondary index are appended
by the primary key value, if a query involves only columns from
the primary key and a secondary index, the latter is considered
to be covering.

That tricks mysql_update() to mark for reading only columns
from the secondary index when it does an index scan to retrieve
rows to update in case a part of that key is also being
updated. However, there may be other columns in WHERE that are
part of the primary key, but not the secondary one.

What we actually want to do in this case is to add index
columns to the existing WHERE columns bitmap rather than
replace it.

mysql-test/r/innodb_mysql.result:
  Test case for bug #53830.
mysql-test/t/innodb_mysql.test:
  Test case for bug #53830.
sql/sql_update.cc:
  Add index columns to the read_set bitmap, don't replace it.
sql/table.cc:
  Added a new add_read_columns_used_by_index() function to 
  st_table.
sql/table.h:
  Added a new add_read_columns_used_by_index() function to 
  st_table.
 2010-05-25 18:43:45 +04:00
Dmitry Lenev
bee0f214fd Pre-requisite patch for bug #51263 "Deadlock between 
transactional SELECT and ALTER TABLE ... REBUILD PARTITION".

The goal of this patch is to decouple type of metadata
lock acquired for table by open_tables() from type of
table-level lock to be acquired on it.

To achieve this we change approach to how we determine what
type of metadata lock should be acquired on table to be open.
Now instead of inferring it at open_tables() time from flags
and type of table-level lock we rely on that type of metadata
lock is properly set at parsing time and is not changed
further.

sql/ha_ndbcluster.cc:
  Now one needs to properly initialize table list element's
  MDL_request object before calling mysql_rm_table_part2().
sql/lock.cc:
  lock_table_names() no longer initializes table list elements'
  MDL_request objects. Now proper initialization of these
  requests is a responsibility of the caller.
sql/lock.h:
  Removed MYSQL_OPEN_TAKE_UPGRADABLE_MDL flag which became
  unnecessary. Thanks to the fact that we don't reset type of
  requests for metadata locks between re-executions we now can
  figure out that upgradable locks are requested by simply
  looking at their type which were set in the parser. As result
  this flag became redundant.
sql/mdl.h:
  Added version of new operator which simplifies allocation of
  MDL_request objects on a MEM_ROOT.
sql/sp_head.cc:
  Added comment explaining why it is OK to infer type of
  metadata lock to request from type of table-level lock
  for prelocking.
  Added enum_mdl_type argument to sp_add_to_query_tables()
  to simplify its usage in trigger implementation.
sql/sp_head.h:
  Added enum_mdl_type argument to sp_add_to_query_tables()
  to simplify its usage in trigger implementation.
sql/sql_base.cc:
  - open_table_get_mdl_lock():
    Preserve type of MDL_request for table list element which
    was set in the parser by creating MDL_request objects on
    memory root if MYSQL_OPEN_FORCE_SHARED_MDL or
    MYSQL_OPEN_FORCE_SHARED_HIGH_PRIO_MDL flag were specified.
    Thanks to this and to the fact that we no longer reset
    type of requests for metadata locks between re-executions
    we no longer need to acquire exclusive metadata lock on
    table to be created in a special way. This lock is acquired
    by code handling acquiring of upgradable locks.
    Also changed signature/calling convention for this function
    to simplify its usage.
  - Accordingly special lock strategy for table list elements
    which was used for such locks became unnecessary and was
    removed. Other strategies were renamed.
  - Since we no longer have guarantee that MDL_request object
    which were not satisfied due to lock conflict belongs to
    table list element Open_table_context class and its methods
    were extended to remember pointer to MDL_request which has
    caused problem at request_backoff_action() time and use it
    in recover_from_failed_open(). Similar approach is used
    for cases when problem from which we need to recover is
    not related to MDL but to the table itself. In this case
    we store pointer to the element of table list.
  - Changed open_tables()/open_tables_check_upgradable_mdl()/
    open_tables_acquire_upgradable_mdl() not to rely on
    MYSQL_OPEN_TAKE_UPGRADABLE_MDL flag to understand when
    upgradable metadata locks should be acquired and not to
    infer type of MDL lock from type of table-level lock.
    Instead we assume that type of MDL to be acquired was set
    in the parser (we can do this as type of MDL_request is
    no longer reset between re-executions).
sql/sql_class.h:
  Since we no longer have guarantee that MDL_request object
  which were not satisfied due to lock conflict belongs to
  table list element Open_table_context class and its methods
  were extended to remember pointer to MDL_request which has
  caused problem at request_backoff_action() time and use it
  in recover_from_failed_open(). Similar approach is used
  for cases when problem from which we need to recover is
  not related to MDL but to the table itself. In this case
  we store pointer to the element of table list.
sql/sql_db.cc:
  Now one needs to properly initialize table list element's
  MDL_request object before calling mysql_rm_table_part2()
  or mysql_rename_tables().
sql/sql_lex.cc:
  st_select_lex/st_select_lex_node::add_table_to_list() method
  now has argument which allows specify type of metadata lock
  to be requested for table list element being added.
sql/sql_lex.h:
  - st_select_lex/st_select_lex_node::add_table_to_list()
    method now has argument which specifies type of metadata
    lock to be requested for table list element being added.
    This allows to explicitly set type of MDL lock to be
    acquired for a DDL statement in parser. It is also more
    future-proof than inferring type of MDL request from type
    of table-level lock.
  - Added Yacc_state::m_mdl_type member which specifies which
    type of metadata lock should be requested for tables to be
    added to table list by a grammar rule in cases when the same
    rule is used in several statements requiring different kinds
    of metadata locks.
sql/sql_parse.cc:
  - st_select_lex::add_table_to_list() method now has argument
    which specifies type of metadata lock to be requested for
    table list element being added. This allows to explicitly
    set type of MDL lock to be acquired for a DDL statement in
    parser. It is also more future-proof than inferring type of
    MDL request from type of table-level lock.
  - EXCLUSIVE_DOWNGRADABLE_MDL lock strategy has a new name -
    OTLS_DOWNGRADE_IF_EXISTS.
  - Adjusted LOCK TABLES implementation to the fact that we no
    longer infer type of metadata lock to be acquired from table
    level lock and that type of MDL request is set at parsing.
    And thus MYSQL_OPEN_TAKE_UPGRADABLE_MDL flag became
    unnecessary.
sql/sql_prepare.cc:
  TABLE_LIST's lock strategy SHARED_MDL was renamed to OTLS_NONE
  as now it means that metadata lock should not be changed during
  call to open_table() (if it has been already acquired) and is
  also used for exclusive metadata lock.
sql/sql_show.cc:
  st_select_lex::add_table_to_list() method now has argument
  which specifies type of metadata lock to be requested for
  table list element being added.
sql/sql_table.cc:
  - Adjusted mysql_admin_table()'s code to the fact that
    open_tables() no longer determines what kind of metadata
    lock should be obtained basing on type of table-level
    lock and flags. Instead type of metadata lock for table
    to be open should be set before calling open_tables().
  - Changed mysql_alter_table() code to the facts:
    a) that now it is responsibility of caller to properly
    initalize MDL_request in table list elements before calling
    lock_table_names()
    b) and that MYSQL_OPEN_TAKE_UPGRADABLE_MDL is no longer
    necessary since type of metadata lock to be obtained
    at open_tables() time is set during parsing.
  - Changed code of mysql_recreate_table() to properly set
    type of metadata and table-level lock to be obtained
    by mysql_alter_table() which it calls.
sql/sql_trigger.cc:
  Instead of relying on MYSQL_OPEN_TAKE_UPGRADABLE_MDL flag to
  force open_tables() to take an upgradable lock we now specify
  exact type of lock to be taken when constructing table list
  element for table to be open for CREATE/DROP TRIGGER.
sql/sql_view.cc:
  We no longer use TABLE_LIST::EXCLUSIVE_MDL strategy to force
  open_tables() to take an exclusive metadata lock on view to
  be created. Instead we rely on parser setting proper type of
  metadata lock to request and open_tables() acquiring it.
  This became possible thanks to the fact that we no longer
  reset type of MDL_request between statement re-executions.
sql/sql_yacc.yy:
  Instead of inferring type of MDL_request for table to be
  open from type of table-level lock and flags passed to
  open_tables() we now explicitly specify them at parsing.
  This became possible thanks to the fact that we no longer
  reset type of MDL_request between statement re-executions.
  In future this should allow to decouple type of metadata
  lock from type of table-level lock.
  The only exception to this approach is statements implemented
  through mysql_admin_table() which re-uses same table list
  element several times with different types of table-level
  and metadata locks.
  We now also properly initialize MDL_request objects for table
  list elements which are later passed to lock_table_names()
  function.
sql/table.cc:
  Do not reset type of MDL_request between statement
  re-executions. This became unnecessesary as we no longer
  change type of MDL_request residing in table list element.
  In its turn this change allows to set type of MDL_request
  only once - at parsing time.
sql/table.h:
  Got rid of TABLE_LIST::EXCLUSIVE_MDL lock strategy.
  Now we can specify that we need to acquire exclusive lock
  on table to be processed by open_tables() through setting
  an appropriate type of MDL_request at parsing time (this
  became possible thanks to the fact that we no longer reset
  types of MDL_request's belonging to table list elements
  between statement re-execution).
  Strategy SHARED_MDL was renamed to OTLS_NONE as now it
  means that metadata lock should not be changed during call
  to open_table() (if it has been already acquired) and is
  also used for exclusive metadata lock.
  Strategy EXCLUSIVE_DOWNGRADABLE_MDL was renamed to
  OTLS_DOWNGRADE_IF_EXISTS.
 2010-05-25 16:35:01 +04:00
Mattias Jonsson
7af0bbc3bc Merge of bug#51851. 
Also moved HA_DATA_PARTITION from ha_partition.cc to table.h.
 2010-05-24 14:51:59 +02:00
Alexey Kopytov
c7fbc8c67c Manual merge from mysql-5.1-bugteam to mysql-trunk-merge. 
Text conflict in sql/log.cc
Text conflict in sql/log.h
Text conflict in sql/sql_class.cc
Text conflict in sql/sql_load.cc
 2010-05-11 12:23:57 +04:00
Alexey Kopytov
5ef2bdea81 Manual merge of mysql-5.1-bugteam to mysql-trunk-merge. 
Conflicts:

Text conflict in mysql-test/r/grant.result
Text conflict in mysql-test/t/grant.test
Text conflict in mysys/mf_loadpath.c
Text conflict in sql/slave.cc
Text conflict in sql/sql_priv.h
 2010-05-09 02:03:35 +04:00
He Zhenxing
df0b6707da Bug#53189 Table map version unused and can be removed 
MYSQL_BIN_LOG m_table_map_version member and it's associated 
functions were not used in the logic of binlogging and replication,
this patch removed all related code.

sql/log.cc:
  removed unused m_table_map_version variable and functions
sql/log.h:
  removed unused m_table_map_version variable and functions
sql/log_event.h:
  Removed unused LOG_EVENT_UPDATE_TABLE_MAP_VERSION_F flag
sql/sql_class.cc:
  Removed unused LOG_EVENT_UPDATE_TABLE_MAP_VERSION_F flag
sql/sql_load.cc:
  Removed unused LOG_EVENT_UPDATE_TABLE_MAP_VERSION_F flag
sql/table.cc:
  removed unused table_map_version variable
sql/table.h:
  removed unused table_map_version variable
 2010-05-08 11:25:33 +08:00
Martin Hansson
0c82d3f3f4 Merge of fix for Bug#52357 2010-05-06 10:59:28 +02:00
Martin Hansson
1eada91053 Bug#52357: Assertion failed: join->best_read in 
greedy_search optimizer_search_depth=0

The algorithm inside restore_prev_nj_state failed to
properly update the counters within the NESTED_JOIN
tree. The counter was decremented each time a table in the
node was removed from the QEP, the correct thing to do being
only to decrement it when the last table in the child node
was removed from the plan. This lead to node counters
getting negative values and the plan thus appeared
impossible. An assertion caught this.

Fixed by not recursing up the tree unless the last table in
the join nest node is removed from the plan
 2010-05-06 10:45:00 +02:00
Alexander Nozdrin
60333fbec9 A post-fix patch for WL#4877/WL#5030: 
Fix tons of warnings about mismatch struct and class usage.
 2010-04-12 17:17:37 +04:00
Mats Kindahl
46bd78b9ee WL#5030: Splitting mysql_priv.h 
Adding my_global.h first in all files using
NO_EMBEDDED_ACCESS_CHECKS.

Correcting a merge problem resulting from a changed definition
of check_some_access compared to the original patches.
 2010-04-07 13:58:40 +02:00
Mats Kindahl
23d8586dbf WL#5030: Split and remove mysql_priv.h 
This patch:

- Moves all definitions from the mysql_priv.h file into
  header files for the component where the variable is
  defined
- Creates header files if the component lacks one
- Eliminates all include directives from mysql_priv.h
- Eliminates all circular include cycles
- Rename time.cc to sql_time.cc
- Rename mysql_priv.h to sql_priv.h
 2010-03-31 16:05:33 +02:00
Mattias Jonsson
47906997c7 Bug#51851: Server with SBR locks mutex twice on LOAD DATA 
into partitioned MyISAM table

Problem was that the ha_data structure was introduced in 5.1
and only used for partitioning first, but with the intention
of be of use for others engines as well, and when used by other
engines it would clash if it also was partitioned.

Solution is to move the partitioning specific data to a separate
structure, with its own mutex (which is used for auto_increment).

Also did rename PARTITION_INFO to PARTITION_STATS since there
already exist a class named partition_info, also cleaned up
some related variables.

mysql-test/r/partition_binlog_stmt.result:
  Bug#51851: Server with SBR locks mutex twice on LOAD DATA
  into partitioned MyISAM table
  
  New result file
mysql-test/t/partition_binlog_stmt.test:
  Bug#51851: Server with SBR locks mutex twice on LOAD DATA
  into partitioned MyISAM table
  
  New result file
sql/ha_ndbcluster.cc:
  Bug#51851: Server with SBR locks mutex twice on LOAD DATA
  into partitioned MyISAM table
  
  Rename of PARTITION_INFO to PARTITION_STATS to better
  match the use (and there is also a class named
  partition_info...)
sql/ha_ndbcluster.h:
  Bug#51851: Server with SBR locks mutex twice on LOAD DATA
  into partitioned MyISAM table
  
  Rename of PARTITION_INFO to PARTITION_STATS to better
  match the use (and there is also a class named
  partition_info...)
sql/ha_partition.cc:
  Bug#51851: Server with SBR locks mutex twice on LOAD DATA
  into partitioned MyISAM table
  
  Removed the partitioning engines use of ha_data in
  TABLE_SHARE and added ha_part_data instead, since
  they collide if used in the same time.
  
  Rename of PARTITION_INFO to PARTITION_STATS to better
  match the use (and there is also a class named
  partition_info...)
  
  Removed some dead code.
sql/ha_partition.h:
  Bug#51851: Server with SBR locks mutex twice on LOAD DATA
  into partitioned MyISAM table
  
  Removed some dead code.
  
  Rename of PARTITION_INFO to PARTITION_STATS to better
  match the use (and there is also a class named
  partition_info...)
  
  Removed the partitioning engines use of ha_data in
  TABLE_SHARE and added ha_part_data instead, since
  they collide if used in the same time.
sql/handler.cc:
  Bug#51851: Server with SBR locks mutex twice on LOAD DATA
  into partitioned MyISAM table
  
  Rename of PARTITION_INFO to PARTITION_STATS to better
  match the use (and there is also a class named
  partition_info...)
sql/handler.h:
  Bug#51851: Server with SBR locks mutex twice on LOAD DATA
  into partitioned MyISAM table
  
  Rename of PARTITION_INFO to PARTITION_STATS to better
  match the use (and there is also a class named
  partition_info...)
sql/mysql_priv.h:
  Bug#51851: Server with SBR locks mutex twice on LOAD DATA
  into partitioned MyISAM table
  
  Removed the partitioning engines use of ha_data in
  TABLE_SHARE and added ha_part_data instead, since
  they collide if used in the same time.
  
  Added key_PARTITION_LOCK_auto_inc for instrumentation.
sql/mysqld.cc:
  Bug#51851: Server with SBR locks mutex twice on LOAD DATA
  into partitioned MyISAM table
  
  Removed the partitioning engines use of ha_data in
  TABLE_SHARE and added ha_part_data instead, since
  they collide if used in the same time.
  
  Added key_PARTITION_LOCK_auto_inc for instrumentation.
sql/partition_info.h:
  Bug#51851: Server with SBR locks mutex twice on LOAD DATA
  into partitioned MyISAM table
  
  Removed part_state* since it was not in use.
sql/sql_partition.cc:
  Bug#51851: Server with SBR locks mutex twice on LOAD DATA
  into partitioned MyISAM table
  
  Removed part_state* since it was not in use.
sql/sql_partition.h:
  Bug#51851: Server with SBR locks mutex twice on LOAD DATA
  into partitioned MyISAM table
  
  Cleaned up old commented out code.
  
  Removed part_state* since it was not in use.
sql/sql_show.cc:
  Bug#51851: Server with SBR locks mutex twice on LOAD DATA
  into partitioned MyISAM table
  
  Rename of PARTITION_INFO to PARTITION_STATS to better
  match the use (and there is also a class named
  partition_info...)
  
  Renamed partition_info to partition_info_str, since
  partition_info is a name of a class.
sql/sql_table.cc:
  Bug#51851: Server with SBR locks mutex twice on LOAD DATA
  into partitioned MyISAM table
  
  Renamed partition_info to partition_info_str, since
  partition_info is a name of a class.
sql/table.cc:
  Bug#51851: Server with SBR locks mutex twice on LOAD DATA
  into partitioned MyISAM table
  
  Removed the partitioning engines use of ha_data in
  TABLE_SHARE and added ha_part_data instead, since
  they collide if used in the same time.
  
  Renamed partition_info to partition_info_str, since
  partition_info is a name of a class.
  
  removed part_state* since it was not in use.
sql/table.h:
  Bug#51851: Server with SBR locks mutex twice on LOAD DATA
  into partitioned MyISAM table
  
  Removed the partitioning engines use of ha_data in
  TABLE_SHARE and added ha_part_data instead, since
  they collide if used in the same time.
  
  Renamed partition_info to partition_info_str, since
  partition_info is a name of a class.
  
  removed part_state* since it was not in use.
 2010-03-30 22:52:45 +02:00
Alexander Nozdrin
b3018e8613 Manual merge from mysql-trunk-merge. 
Conflicts:
  - client/mysql.cc
  - client/mysqldump.c
  - configure.in
  - mysql-test/r/csv.result
  - mysql-test/r/func_time.result
  - mysql-test/r/show_check.result
  - mysql-test/r/sp-error.result
  - mysql-test/r/sp.result
  - mysql-test/r/sp_trans.result
  - mysql-test/r/type_blob.result
  - mysql-test/r/type_timestamp.result
  - mysql-test/r/warnings.result
  - mysql-test/suite/rpl/r/rpl_sp.result
  - sql/mysql_priv.h
  - sql/mysqld.cc
  - sql/sp.cc
  - sql/sql_base.cc
  - sql/sql_table.cc
  - sql/sql_trigger.cc
  - sql/sql_view.cc
  - sql/table.h
  - sql/share/errmsg.txt
  - mysql-test/suite/sys_vars/r/log_bin_trust_routine_creators_basic.result
 2010-02-24 16:52:27 +03:00
Alexey Kopytov
0afa209018 Manual merge of mysql-5.1-bugteam to mysql-trunk-merge. 2010-02-22 00:33:11 +03:00
Konstantin Osipov
c8555bdb35 Merge next-mr -> next-4284 2010-02-03 16:43:03 +03:00
Konstantin Osipov
665100b69d Merge next-mr -> next-4284. 2010-02-02 02:22:16 +03:00
Dmitry Lenev
eba5d30e67 Implement new type-of-operation-aware metadata locks. 
Add a wait-for graph based deadlock detector to the
MDL subsystem.

Fixes bug #46272 "MySQL 5.4.4, new MDL: unnecessary deadlock" and
bug #37346 "innodb does not detect deadlock between update and
alter table".

The first bug manifested itself as an unwarranted abort of a
transaction with ER_LOCK_DEADLOCK error by a concurrent ALTER
statement, when this transaction tried to repeat use of a
table, which it has already used in a similar fashion before
ALTER started.

The second bug showed up as a deadlock between table-level
locks and InnoDB row locks, which was "detected" only after
innodb_lock_wait_timeout timeout.

A transaction would start using the table and modify a few
rows.
Then ALTER TABLE would come in, and start copying rows
into a temporary table. Eventually it would stumble on
the modified records and get blocked on a row lock.
The first transaction would try to do more updates, and get
blocked on thr_lock.c lock.
This situation of circular wait would only get resolved
by a timeout.

Both these bugs stemmed from inadequate solutions to the
problem of deadlocks occurring between different
locking subsystems.

In the first case we tried to avoid deadlocks between metadata
locking and table-level locking subsystems, when upgrading shared
metadata lock to exclusive one.
Transactions holding the shared lock on the table and waiting for
some table-level lock used to be aborted too aggressively.

We also allowed ALTER TABLE to start in presence of transactions
that modify the subject table. ALTER TABLE acquires
TL_WRITE_ALLOW_READ lock at start, and that block all writes
against the table (naturally, we don't want any writes to be lost
when switching the old and the new table). TL_WRITE_ALLOW_READ
lock, in turn, would block the started transaction on thr_lock.c
lock, should they do more updates. This, again, lead to the need
to abort such transactions.

The second bug occurred simply because we didn't have any
mechanism to detect deadlocks between the table-level locks
in thr_lock.c and row-level locks in InnoDB, other than
innodb_lock_wait_timeout.

This patch solves both these problems by moving lock conflicts
which are causing these deadlocks into the metadata locking
subsystem, thus making it possible to avoid or detect such
deadlocks inside MDL.

To do this we introduce new type-of-operation-aware metadata
locks, which allow MDL subsystem to know not only the fact that
transaction has used or is going to use some object but also what
kind of operation it has carried out or going to carry out on the
object.

This, along with the addition of a special kind of upgradable
metadata lock, allows ALTER TABLE to wait until all
transactions which has updated the table to go away.
This solves the second issue.
Another special type of upgradable metadata lock is acquired
by LOCK TABLE WRITE. This second lock type allows to solve the
first issue, since abortion of table-level locks in event of
DDL under LOCK TABLES becomes also unnecessary.

Below follows the list of incompatible changes introduced by
this patch:

- From now on, ALTER TABLE and CREATE/DROP TRIGGER SQL (i.e. those
  statements that acquire TL_WRITE_ALLOW_READ lock)
  wait for all transactions which has *updated* the table to
  complete.

- From now on, LOCK TABLES ... WRITE, REPAIR/OPTIMIZE TABLE
  (i.e. all statements which acquire TL_WRITE table-level lock) wait
  for all transaction which *updated or read* from the table
  to complete.
  As a consequence, innodb_table_locks=0 option no longer applies
  to LOCK TABLES ... WRITE.

- DROP DATABASE, DROP TABLE, RENAME TABLE no longer abort
  statements or transactions which use tables being dropped or
  renamed, and instead wait for these transactions to complete.

- Since LOCK TABLES WRITE now takes a special metadata lock,
  not compatible with with reads or writes against the subject table
  and transaction-wide, thr_lock.c deadlock avoidance algorithm
  that used to ensure absence of deadlocks between LOCK TABLES
  WRITE and other statements is no longer sufficient, even for
  MyISAM. The wait-for graph based deadlock detector of MDL
  subsystem may sometimes be necessary and is involved. This may
  lead to ER_LOCK_DEADLOCK error produced for multi-statement
  transactions even if these only use MyISAM:

  session 1:         session 2:
  begin;

  update t1 ...      lock table t2 write, t1 write;
                     -- gets a lock on t2, blocks on t1

  update t2 ...
  (ER_LOCK_DEADLOCK)

- Finally,  support of LOW_PRIORITY option for LOCK TABLES ... WRITE
  was abandoned.
  LOCK TABLE ... LOW_PRIORITY WRITE from now on has the same
  priority as the usual LOCK TABLE ... WRITE.
  SELECT HIGH PRIORITY no longer trumps LOCK TABLE ... WRITE  in
  the wait queue.

- We do not take upgradable metadata locks on implicitly
  locked tables. So if one has, say, a view v1 that uses
  table t1, and issues:
  LOCK TABLE v1 WRITE;
  FLUSH TABLE t1; -- (or just 'FLUSH TABLES'),
  an error is produced.
  In order to be able to perform DDL on a table under LOCK TABLES,
  the table must be locked explicitly in the LOCK TABLES list.

mysql-test/include/handler.inc:
  Adjusted test case to trigger an execution path on which bug 41110
  "crash with handler command when used concurrently with alter
  table" and bug 41112 "crash in mysql_ha_close_table/get_lock_data
  with alter table" were originally discovered. Left old test case
  which no longer triggers this execution path for the sake of
  coverage.
  Added test coverage for HANDLER SQL statements and type-aware
  metadata locks.
  Added a test for the global shared lock and HANDLER SQL.
  Updated tests to take into account that the old simple deadlock
  detection heuristics was replaced with a graph-based deadlock
  detector.
mysql-test/r/debug_sync.result:
  Updated results (see debug_sync.test).
mysql-test/r/handler_innodb.result:
  Updated results (see handler.inc test).
mysql-test/r/handler_myisam.result:
  Updated results (see handler.inc test).
mysql-test/r/innodb-lock.result:
  Updated results (see innodb-lock.test).
mysql-test/r/innodb_mysql_lock.result:
  Updated results (see innodb_mysql_lock.test).
mysql-test/r/lock.result:
  Updated results (see lock.test).
mysql-test/r/lock_multi.result:
  Updated results (see lock_multi.test).
mysql-test/r/lock_sync.result:
  Updated results (see lock_sync.test).
mysql-test/r/mdl_sync.result:
  Updated results (see mdl_sync.test).
mysql-test/r/sp-threads.result:
  SHOW PROCESSLIST output has changed due to the fact that waiting
  for LOCK TABLES WRITE now happens within metadata locking
  subsystem.
mysql-test/r/truncate_coverage.result:
  Updated results (see truncate_coverage.test).
mysql-test/suite/funcs_1/datadict/processlist_val.inc:
  SELECT FROM I_S.PROCESSLIST output has changed due to fact that
  waiting for LOCK TABLES WRITE now happens within metadata locking
  subsystem.
mysql-test/suite/funcs_1/r/processlist_val_no_prot.result:
  SELECT FROM I_S.PROCESSLIST output has changed due to fact that
  waiting for LOCK TABLES WRITE now happens within metadata locking
  subsystem.
mysql-test/suite/rpl/t/rpl_sp.test:
  Updated to a new SHOW PROCESSLIST state name.
mysql-test/t/debug_sync.test:
  Use LOCK TABLES READ instead of LOCK TABLES WRITE as the latter
  no longer allows to trigger execution path involving waiting on
  thr_lock.c lock and therefore reaching debug sync-point covered
  by this test.
mysql-test/t/innodb-lock.test:
  Adjusted test case to the fact that innodb_table_locks=0 option is
  no longer supported, since LOCK TABLES WRITE handles all its
  conflicts within MDL subsystem.
mysql-test/t/innodb_mysql_lock.test:
  Added test for bug #37346 "innodb does not detect deadlock between
  update and alter table".
mysql-test/t/lock.test:
  Added test coverage which checks the fact that we no longer support
  DDL under LOCK TABLES on tables which were locked implicitly.
  Adjusted existing test cases accordingly.
mysql-test/t/lock_multi.test:
  Added test for bug #46272 "MySQL 5.4.4, new MDL: unnecessary
  deadlock".  Adjusted other test cases to take into account the
  fact that waiting for LOCK TABLES ... WRITE now happens within MDL
  subsystem.
mysql-test/t/lock_sync.test:
  Since LOCK TABLES ... WRITE now takes SNRW metadata lock for
  tables locked explicitly we have to implicitly lock InnoDB tables
  (through view) to trigger the table-level lock conflict between
  TL_WRITE and TL_WRITE_ALLOW_WRITE.
mysql-test/t/mdl_sync.test:
  Added basic test coverage for type-of-operation-aware metadata
  locks. Also covered with tests some use cases involving HANDLER
  statements in which a deadlock could arise.
  Adjusted existing tests to take type-of-operation-aware MDL into
  account.
mysql-test/t/multi_update.test:
  Update to a new SHOW PROCESSLIST state name.
mysql-test/t/truncate_coverage.test:
  Adjusted test case after making LOCK TABLES WRITE to wait until
  transactions that use the table to be locked are completed.
  Updated to the changed name of DEBUG_SYNC point.
sql/handler.cc:
  Global read lock functionality has been
  moved into a class.
sql/lock.cc:
  Global read lock functionality has been
  moved into a class.
  Updated code to use the new MDL API.
sql/mdl.cc:
  Introduced new type-of-operation aware metadata locks.
  To do this:
  - Changed MDL_lock to use one list for waiting requests and one
    list for granted requests. For each list, added a bitmap
    that holds information what lock types a list contains.
    Added a helper class MDL_lock::List to manipulate with granted
    and waited lists while keeping the bitmaps in sync
    with list contents.
  - Changed lock-compatibility functions to use bitmaps that
    define compatibility.
  - Introduced a graph based deadlock detector inspired by
    waiting_threads.c from Maria implementation.
  - Now that we have a deadlock detector, and no longer have
    a global lock to protect individual lock objects, but rather
    use an rw lock per object, removed redundant code for upgrade,
    and the global read lock. Changed the MDL API to
    no longer require the caller to acquire the global
    intention exclusive lock by means of a separate method.
    Removed a few more methods that became redundant.
  - Removed deadlock detection heuristic, it has been made
    obsolete by the deadlock detector.
  - With operation-type-aware metadata locks, MDL subsystem has
    become aware of potential conflicts between DDL and open
    transactions. This made it possible to remove calls to
    mysql_abort_transactions_with_shared_lock() from acquisition
    paths for exclusive lock and lock upgrade. Now we can simply
    wait for these transactions to complete without fear of
    deadlock. Function mysql_lock_abort() has also become
    unnecessary for all conflicting cases except when a DDL
    conflicts with a connection that has an open HANDLER.
sql/mdl.h:
  Introduced new type-of-operation aware metadata locks.
  Introduced a graph based deadlock detector and supporting
  methods.
  Added comments.
  God rid of redundant API calls.
  Renamed m_lt_or_ha_sentinel to m_trans_sentinel,
  since now it guards the global read lock as well as
  LOCK TABLES and HANDLER locks.
sql/mysql_priv.h:
  Moved the global read lock functionality into a
  class.
  Added MYSQL_OPEN_FORCE_SHARED_MDL flag which forces
  open_tables() to take MDL_SHARED on tables instead of
  metadata locks specified in the parser. We use this to
  allow PREPARE run concurrently in presence of
  LOCK TABLES ... WRITE.
  Added signature for find_table_for_mdl_ugprade().
sql/set_var.cc:
  Global read lock functionality has been
  moved into a class.
sql/sp_head.cc:
  When creating TABLE_LIST elements for prelocking or
  system tables set the type of request for metadata
  lock according to the operation that will be performed
  on the table.
sql/sql_base.cc:
  - Updated code to use the new MDL API.
  - In order to avoid locks starvation we take upgradable
    locks all at once. As result implicitly locked tables no
    longer get an upgradable lock. Consequently DDL and FLUSH
    TABLES for such tables is prohibited.
    find_write_locked_table() was replaced by
    find_table_for_mdl_upgrade() function.
    open_table() was adjusted to return TABLE instance with
    upgradable ticket when necessary.
  - We no longer wait for all locks on OT_WAIT back off
    action -- only on the lock that caused the wait
    conflict. Moreover, now we distinguish cases when we
    have to wait due to conflict in MDL and old version
    of table in TDC.
  - Upate mysql_notify_threads_having_share_locks()
    to only abort thr_lock.c waits of threads that
    have open HANDLERs, since lock conflicts with only
    these threads now can lead to deadlocks not detectable
    by the MDL deadlock detector.
  - Remove mysql_abort_transactions_with_shared_locks()
    which is no longer needed.
sql/sql_class.cc:
  Global read lock functionality has been moved into a class.
  Re-arranged code in THD::cleanup() to simplify assert.
sql/sql_class.h:
  Introduced class to incapsulate global read lock
  functionality.
  Now sentinel in MDL subsystem guards the global read lock
  as well as LOCK TABLES and HANDLER locks. Adjusted code
  accordingly.
sql/sql_db.cc:
  Global read lock functionality has been moved into a class.
sql/sql_delete.cc:
  We no longer acquire upgradable metadata locks on tables
  which are locked by LOCK TABLES implicitly. As result
  TRUNCATE TABLE is no longer allowed for such tables.
  Updated code to use the new MDL API.
sql/sql_handler.cc:
  Inform MDL_context about presence of open HANDLERs.
  Since HANLDERs break MDL protocol by acquiring table-level
  lock while holding only S metadata lock on a table MDL
  subsystem should take special care about such contexts (Now
  this is the only case when mysql_lock_abort() is used).
sql/sql_parse.cc:
  Global read lock functionality has been moved into a class.
  Do not take upgradable metadata locks when opening tables
  for CREATE TABLE SELECT as it is not necessary and limits
  concurrency.
  When initializing TABLE_LIST objects before adding them
  to the table list set the type of request for metadata lock
  according to the operation that will be performed on the
  table.
  We no longer acquire upgradable metadata locks on tables
  which are locked by LOCK TABLES implicitly. As result FLUSH
  TABLES is no longer allowed for such tables.
sql/sql_prepare.cc:
  Use MYSQL_OPEN_FORCE_SHARED_MDL flag when opening
  tables during PREPARE. This allows PREPARE to run
  concurrently in presence of LOCK TABLES ... WRITE.
sql/sql_rename.cc:
  Global read lock functionality has been moved into a class.
sql/sql_show.cc:
  Updated code to use the new MDL API.
sql/sql_table.cc:
  Global read lock functionality has been moved into a class.
  We no longer acquire upgradable metadata locks on tables
  which are locked by LOCK TABLES implicitly. As result DROP
  TABLE is no longer allowed for such tables.
  Updated code to use the new MDL API.
sql/sql_trigger.cc:
  Global read lock functionality has been moved into a class.
  We no longer acquire upgradable metadata locks on tables
  which are locked by LOCK TABLES implicitly. As result
  CREATE/DROP TRIGGER is no longer allowed for such tables.
  Updated code to use the new MDL API.
sql/sql_view.cc:
  Global read lock functionality has been moved into a class.
  Fixed results of wrong merge that led to misuse of GLR API.
  CREATE VIEW statement is not a commit statement.
sql/table.cc:
  When resetting TABLE_LIST objects for PS or SP re-execution
  set the type of request for metadata lock according to the
  operation that will be performed on the table. Do the same
  in auxiliary function initializing metadata lock requests
  in a table list.
sql/table.h:
  When initializing TABLE_LIST objects set the type of request
  for metadata lock according to the operation that will be
  performed on the table.
sql/transaction.cc:
  Global read lock functionality has been moved into a class.
 2010-02-01 14:43:06 +03:00
Sergey Vojtovich
0897669cba BUG#49902 - SELECT returns incorrect results 
Queries optimized with GROUP_MIN_MAX didn't cleanup KEYREAD
optimization properly. As a result subsequent queries may
return incomplete rows (fields are initialized to default
values).

mysql-test/r/group_min_max.result:
  A test case for BUG#49902.
mysql-test/t/group_min_max.test:
  A test case for BUG#49902.
sql/opt_range.cc:
  Refactor of KEYREAD optimization switch so that KEYREAD
  handler state is in sync with st_table::key_read flag.
  
  All SQL code is supposed to switch KEYREAD optimization
  via st_table::set_keyread().
sql/opt_sum.cc:
  Refactor of KEYREAD optimization switch so that KEYREAD
  handler state is in sync with st_table::key_read flag.
  
  All SQL code is supposed to switch KEYREAD optimization
  via st_table::set_keyread().
sql/sql_select.cc:
  Refactor of KEYREAD optimization switch so that KEYREAD
  handler state is in sync with st_table::key_read flag.
  
  All SQL code is supposed to switch KEYREAD optimization
  via st_table::set_keyread().
sql/sql_update.cc:
  Refactor of KEYREAD optimization switch so that KEYREAD
  handler state is in sync with st_table::key_read flag.
  
  All SQL code is supposed to switch KEYREAD optimization
  via st_table::set_keyread().
sql/table.cc:
  Refactor of KEYREAD optimization switch so that KEYREAD
  handler state is in sync with st_table::key_read flag.
  
  All SQL code is supposed to switch KEYREAD optimization
  via st_table::set_keyread().
sql/table.h:
  Refactor of KEYREAD optimization switch so that KEYREAD
  handler state is in sync with st_table::key_read flag.
  
  All SQL code is supposed to switch KEYREAD optimization
  via st_table::set_keyread().
 2010-02-09 12:53:13 +04:00
Marc Alff
3d91522561 WL#2360 Performance schema 
Part IV: sql instrumentation
 2010-01-06 22:42:07 -07:00
Konstantin Osipov
92b1c2f3ca Merge next-mr -> next-4284. 2009-12-15 22:03:56 +03:00
Alexander Nozdrin
aceea2342f Manual merge from mysql-trunk-merge. 
Conflicts:
  - extra/comp_err.c
  - mysql-test/collections/default.experimental
  - mysql-test/r/archive.result
  - mysql-test/r/select.result
  - mysql-test/suite/binlog/r/binlog_unsafe.result
  - mysql-test/suite/binlog/t/binlog_unsafe.test
  - mysql-test/suite/rpl/t/disabled.def
  - mysql-test/t/archive.test
  - mysql-test/t/select.test
  - sql/item.cc
  - sql/item.h
  - sql/item_timefunc.cc
  - sql/sql_base.cc
  - sql/sql_delete.cc
  - sql/sql_load.cc
  - sql/sql_partition.cc
  - sql/sql_table.cc
  - storage/innobase/handler/ha_innodb.cc
  - vio/vio.c
 2009-12-12 23:38:59 +03:00
Alexander Nozdrin
5676713687 Manual merge from mysql-trunk. 
Conflicts:
  - client/mysqltest.cc
  - mysql-test/collections/default.experimental
  - mysql-test/suite/rpl/t/disabled.def
  - sql/mysqld.cc
  - sql/opt_range.cc
  - sql/sp.cc
  - sql/sql_acl.cc
  - sql/sql_partition.cc
  - sql/sql_table.cc
 2009-12-11 12:39:38 +03:00
Jon Olav Hauglid
fb6b5ee462 Backport of revno: 2617.68.37 
Bug #46654 False deadlock on concurrent DML/DDL with partitions, 
           inconsistent behavior

The problem was that if one connection is running a multi-statement 
transaction which involves a single partitioned table, and another 
connection attempts to alter the table, the first connection gets 
ER_LOCK_DEADLOCK and cannot proceed anymore, even when the ALTER TABLE 
statement in another connection has timed out or failed.

The reason for this was that the prepare phase for ALTER TABLE for 
partitioned tables removed all instances of the table from the table 
definition cache before it started waiting on the lock. The transaction 
running in the first connection would notice this and report ER_LOCK_DEADLOCK. 

This patch changes the prep_alter_part_table() ALTER TABLE code so that 
tdc_remove_table() is no longer called. Instead, only the TABLE instance
changed by prep_alter_part_table() is marked as needing reopen.

The patch also removes an unnecessary call to tdc_remove_table() from 
mysql_unpack_partition() as the changed TABLE object is destroyed by the 
caller at a later point.

Test case added in partition_sync.test.
 2009-12-10 14:26:00 +01:00
Jon Olav Hauglid
5e1dfa4c06 Backport of revno: 2617.71.1 
Bug#42546 Backup: RESTORE fails, thinking it finds an existing table

The problem occured when a MDL locking conflict happened for a non-existent 
table between a CREATE and a INSERT statement. The code for CREATE 
interpreted this lock conflict to mean that the table existed, 
which meant that the statement failed when it should not have.
The problem could occur for CREATE TABLE, CREATE TABLE LIKE and
ALTER TABLE RENAME.

This patch fixes the problem for CREATE TABLE and CREATE TABLE LIKE.
It is based on code backported from the mysql-6.1-fk tree written
by Dmitry Lenev. CREATE now uses normal open_and_lock_tables() code 
to acquire exclusive locks. This means that for the test case in the bug 
description, CREATE will wait until INSERT completes so that it can 
get the exclusive lock. This resolves the reported bug.

The patch also prohibits CREATE TABLE and CREATE TABLE LIKE under 
LOCK TABLES. Note that this is an incompatible change and must 
be reflected in the documentation. Affected test cases have been
updated.

mdl_sync.test contains tests for CREATE TABLE and CREATE TABLE LIKE.

Fixing the issue for ALTER TABLE RENAME is beyond the scope of this
patch. ALTER TABLE cannot be prohibited from working under LOCK TABLES
as this could seriously impact customers and a proper fix would require
a significant rewrite.
 2009-12-10 11:53:20 +01:00
Konstantin Osipov
f26f632b44 Backport of: 
------------------------------------------------------------
revno: 2617.68.25
committer: Dmitry Lenev <dlenev@mysql.com>
branch nick: mysql-next-bg-pre2-2
timestamp: Wed 2009-09-16 18:26:50 +0400
message:
  Follow-up for one of pre-requisite patches for fixing bug #30977
  "Concurrent statement using stored function and DROP FUNCTION
  breaks SBR".

  Made enum_mdl_namespace enum part of MDL_key class and removed MDL_
  prefix from the names of enum members. In order to do the latter
  changed name of PROCEDURE symbol to PROCEDURE_SYM (otherwise macro
  which was automatically generated for this symbol conflicted with
  MDL_key::PROCEDURE enum member).
 2009-12-10 11:21:38 +03:00
Jon Olav Hauglid
546e16ebd8 Backport of revno: 2617.69.40 
A pre-requisite patch for Bug#30977 "Concurrent statement using 
stored function and DROP FUNCTION breaks SBR".

This patch changes the MDL API by introducing a namespace for
lock keys: MDL_TABLE for tables and views and MDL_PROCEDURE
for stored procedures and functions. The latter is needed for
the fix for Bug#30977.
 2009-12-09 09:51:20 +01:00
Konstantin Osipov
463dcbb51c Backport of revid 2617.69.21, 2617.69.22, 2617.29.23: 
----------------------------------------------------------
revno: 2617.69.21
committer: Konstantin Osipov <kostja@sun.com>
branch nick: 5.4-4284-1-assert
timestamp: Thu 2009-08-13 20:13:55 +0400
message:
  A fix and a test case for Bug#46610 "MySQL 5.4.4: MyISAM MRG engine crash
  on auto-repair of child".
  Also fixes Bug#42862 "Crash on failed attempt to open a children of a
  merge table".

  MERGE engine needs to extend the global table list
  with TABLE_LIST elements for child tables,
  so that they are opened and locked.
  Previously these table list elements were allocated
  in memory of ha_myisammrg object (MERGE engine handler).
  That would lead to access to freed memory in
  recover_from_failed_open_table_attempt(), which would
  try to recover a MERGE table child (MyISAM table)
  and use for that TABLE_LIST of that child.
  But by the time recover_from_failed_open_table_attempt()
  is invoked, ha_myisammrg object that owns this
  TABLE_LIST may be destroyed, and thus TABLE_LIST
  memory freed.

  The fix is to ensure that TABLE_LIST elements
  that are added to the global table list (lex->query_tables)
  are always allocated in thd->mem_root, which is not
  destroyed until end of execution.

  If previously TABLE_LIST elements were allocated
  at ha_myisammrg::open() (i.e. when the TABLE
  object was created and added to the table cache),
  now they are allocated in ha_myisammrg::add_chidlren_list()
  (i.e. right after "open" of the merge parent in
  open_tables()).
  We still create a list of children names
  at ha_myisammrg::open() to use as a basis
  for creation of TABLE_LISTs, that allows
  to avoid reading the merge handler data
  file on every execution.


mysql-test/r/merge_recover.result:
  Test results for Bug#46610.
mysql-test/t/merge_recover-master.opt:
  Option file for Bug#46610 test (need a new test because
  of that option, which is not tested anywhere else).
mysql-test/t/merge_recover.test:
  Add a test case for Bug#46610.
sql/table.h:
  MERGE child child_def_version is now moved from TABLE_LIST
  to MERGE engine specific data structure.
storage/myisammrg/ha_myisammrg.cc:
  Introduce an auxiliary structure to keep MERGE child name
  and definition version. A list of Mrg_child_def is created
  in ha_myisammrg::open() and reused in ha_myisammrg::add_children_list().
 2009-12-08 16:57:25 +03:00
Konstantin Osipov
ce5c87a3d3 Backport of: 
----------------------------------------------------------
revno: 2617.69.20
committer: Konstantin Osipov <kostja@sun.com>
branch nick: 5.4-4284-1-assert
timestamp: Thu 2009-08-13 18:29:55 +0400
message:
  WL#4284 "Transactional DDL locking"
  A review fix.
  Since WL#4284 implementation separated MDL_request and MDL_ticket,
  MDL_request becamse a utility object necessary only to get a ticket.
  Store it by-value in TABLE_LIST with the intent to merge
  MDL_request::key with table_list->table_name and table_list->db
  in future.
  Change the MDL subsystem to not require MDL_requests to
  stay around till close_thread_tables().
  Remove the list of requests from the MDL context.
  Requests for shared metadata locks acquired in open_tables()
  are only used as a list in recover_from_failed_open_table_attempt(),
  which calls mdl_context.wait_for_locks() for this list.
  To keep such list for recover_from_failed_open_table_attempt(),
  introduce a context class (Open_table_context), that collects
  all requests.
  A lot of minor cleanups and simplications that became possible
  with this change.


sql/event_db_repository.cc:
  Remove alloc_mdl_requests(). Now MDL_request instance is a member
  of TABLE_LIST, and init_one_table() initializes it.
sql/ha_ndbcluster_binlog.cc:
  Remove now unnecessary declaration and initialization
  of binlog_mdl_request.
sql/lock.cc:
  No need to allocate MDL requests in lock_table_names() now.
sql/log.cc:
  Use init_one_table() method, remove alloc_mdl_requests(),
  which is now unnecessary.
sql/log_event.cc:
  No need to allocate mdl_request separately now.
  Use init_one_table() method.
sql/log_event_old.cc:
  Update to the new signature of close_tables_for_reopen().
sql/mdl.cc:
  Update try_acquire_exclusive_lock() to be more easy to use.
  Function lock_table_name_if_not_cached() has been removed.
  Make acquire_shared_lock() signature consistent with
  try_acquire_exclusive_lock() signature.
  Remove methods that are no longer used.
  Update comments.
sql/mdl.h:
  Implement an assignment operator that doesn't
  copy MDL_key (MDL_key::operator= is private and
  should remain private).
  This is a hack to work-around assignment of TABLE_LIST
  by value in several places. Such assignments violate
  encapsulation, since only perform a shallow copy.
  In most cases these assignments are a hack on their own.
sql/mysql_priv.h:
  Update signatures of close_thread_tables() and close_tables_for_reopen().
sql/sp.cc:
  Allocate TABLE_LIST in thd->mem_root.
  Use init_one_table().
sql/sp_head.cc:
  Use init_one_table().
  Remove thd->locked_tables_root, it's no longer needed.
sql/sql_acl.cc:
  Use init_mdl_requests() and init_one_table().
sql/sql_base.cc:
  Update to new signatures of try_acquire_shared_lock() and
  try_acquire_exclusive_lock().
  Remove lock_table_name_if_not_cached().
  Fix a bug in open_ltable() that would not return ER_LOCK_DEADLOCK
  in case of a failed lock_tables() and a multi-statement
  transaction.
  Fix a bug in open_and_lock_tables_derived() that would
  not return ER_LOCK_DEADLOCK in case of a multi-statement
  transaction and a failure of lock_tables().
  Move assignment of enum_open_table_action to a method of Open_table_context, a new class that maintains information
  for backoff actions.
  Minor rearrangements of the code.
  Remove alloc_mdl_requests() in functions that work with system
  tables: instead the patch ensures that callers always initialize
  TABLE_LIST argument.
sql/sql_class.cc:
  THD::locked_tables_root is no more.
sql/sql_class.h:
  THD::locked_tables_root is no more.
  Add a declaration for Open_table_context class.
sql/sql_delete.cc:
  Update to use the simplified MDL API.
sql/sql_handler.cc:
  TABLE_LIST::mdl_request is stored by-value now.
  Ensure that mdl_request.ticket is NULL for every request
  that is passed into MDL, to satisfy MDL asserts.
  @ sql/sql_help.cc
  Function open_system_tables_for_read() no longer initializes
  mdl_requests.
  Move TABLE_LIST::mdl_request initialization closer to
  TABLE_LIST initialization.
sql/sql_help.cc:
  Function open_system_tables_for_read() no longer initializes
  mdl_requests.
  Move TABLE_LIST::mdl_request initialization closer to
  TABLE_LIST initialization.
sql/sql_insert.cc:
  Remove assignment by-value of TABLE_LIST in
  TABLEOP_HOOKS. We can't carry over a granted
  MDL ticket from one table list to another.
sql/sql_parse.cc:
  Change alloc_mdl_requests() -> init_mdl_requests().
  @todo We can remove init_mdl_requests() altogether
  in some places: all places that call add_table_to_list()
  already have mdl requests initialized.
sql/sql_plugin.cc:
  Use init_one_table().
  THD::locked_tables_root is no more.
sql/sql_servers.cc:
  Use init_one_table().
sql/sql_show.cc:
  Update acquire_high_priority_shared_lock() to use
  TABLE_LIST::mdl_request rather than allocate an own.
  Fix get_trigger_table_impl() to use init_one_table(),
  check for out of memory, follow the coding style.
sql/sql_table.cc:
  Update to work with TABLE_LIST::mdl_request by-value.
  Remove lock_table_name_if_not_cached().
  The code that used to delegate to it is quite simple and
  concise without it now.
sql/sql_udf.cc:
  Use init_one_table().
sql/sql_update.cc:
  Update to use the new signature of close_tables_for_reopen().
sql/table.cc:
  Move re-setting of mdl_requests for prepared statements
  and stored procedures from close_thread_tables() to
  reinit_stmt_before_use().
  Change alloc_mdl_requests() to init_mdl_requests().
  init_mdl_requests() is a hack that can't be deleted
  until we don't have a list-aware TABLE_LIST constructor.
  Hopefully its use will be minimal
sql/table.h:
  Change alloc_mdl_requests() to init_mdl_requests()
  TABLE_LIST::mdl_request is stored by value.
sql/tztime.cc:
  We no longer initialize mdl requests in open_system_tables_for*()
  functions. Move this initialization closer to initialization
  of the rest of TABLE_LIST members.
storage/myisammrg/ha_myisammrg.cc:
  Simplify mdl_request initialization.
 2009-12-08 12:57:07 +03:00
Konstantin Osipov
a14bbee5ab Backport of revno ## 2617.31.1, 2617.31.3, 2617.31.4, 2617.31.5, 
2617.31.12, 2617.31.15, 2617.31.15, 2617.31.16, 2617.43.1
- initial changeset that introduced the fix for 
Bug#989 and follow up fixes for all test suite failures
introduced in the initial changeset. 
------------------------------------------------------------
revno: 2617.31.1
committer: Davi Arnaut <Davi.Arnaut@Sun.COM>
branch nick: 4284-6.0
timestamp: Fri 2009-03-06 19:17:00 -0300
message:
Bug#989: If DROP TABLE while there's an active transaction, wrong binlog order
WL#4284: Transactional DDL locking

Currently the MySQL server does not keep metadata locks on
schema objects for the duration of a transaction, thus failing
to guarantee the integrity of the schema objects being used
during the transaction and to protect then from concurrent
DDL operations. This also poses a problem for replication as
a DDL operation might be replicated even thought there are
active transactions using the object being modified.

The solution is to defer the release of metadata locks until
a active transaction is either committed or rolled back. This
prevents other statements from modifying the table for the
entire duration of the transaction. This provides commitment
ordering for guaranteeing serializability across multiple
transactions.

- Incompatible change:

If MySQL's metadata locking system encounters a lock conflict,
the usual schema is to use the try and back-off technique to
avoid deadlocks -- this schema consists in releasing all locks
and trying to acquire them all in one go.

But in a transactional context this algorithm can't be utilized
as its not possible to release locks acquired during the course
of the transaction without breaking the transaction commitments.
To avoid deadlocks in this case, the ER_LOCK_DEADLOCK will be
returned if a lock conflict is encountered during a transaction.

Let's consider an example:

A transaction has two statements that modify table t1, then table
t2, and then commits. The first statement of the transaction will
acquire a shared metadata lock on table t1, and it will be kept
utill COMMIT to ensure serializability.

At the moment when the second statement attempts to acquire a
shared metadata lock on t2, a concurrent ALTER or DROP statement
might have locked t2 exclusively. The prescription of the current
locking protocol is that the acquirer of the shared lock backs off
-- gives up all his current locks and retries. This implies that
the entire multi-statement transaction has to be rolled back.

- Incompatible change:

FLUSH commands such as FLUSH PRIVILEGES and FLUSH TABLES WITH READ
LOCK won't cause locked tables to be implicitly unlocked anymore.


mysql-test/extra/binlog_tests/drop_table.test:
  Add test case for Bug#989.
mysql-test/extra/binlog_tests/mix_innodb_myisam_binlog.test:
  Fix test case to reflect the fact that transactions now hold
  metadata locks for the duration of a transaction.
mysql-test/include/mix1.inc:
  Fix test case to reflect the fact that transactions now hold
  metadata locks for the duration of a transaction.
mysql-test/include/mix2.inc:
  Fix test case to reflect the fact that transactions now hold
  metadata locks for the duration of a transaction.
mysql-test/r/flush_block_commit.result:
  Update test case result (WL#4284).
mysql-test/r/flush_block_commit_notembedded.result:
  Update test case result (WL#4284).
mysql-test/r/innodb.result:
  Update test case result (WL#4284).
mysql-test/r/innodb_mysql.result:
  Update test case result (WL#4284).
mysql-test/r/lock.result:
  Add test case result for an effect of WL#4284/Bug#989
  (all locks should be released when a connection terminates).
mysql-test/r/mix2_myisam.result:
  Update test case result (effects of WL#4284/Bug#989).
mysql-test/r/not_embedded_server.result:
  Update test case result (effects of WL#4284/Bug#989).
  Add a test case for interaction of WL#4284 and FLUSH PRIVILEGES.
mysql-test/r/partition_innodb_semi_consistent.result:
  Update test case result (effects of WL#4284/Bug#989).
mysql-test/r/partition_sync.result:
  Temporarily disable the test case for Bug#43867,
  which will be fixed by a subsequent backport.
mysql-test/r/ps.result:
  Add a test case for effect of PREPARE on transactional
  locks: we take a savepoint at beginning of PREAPRE
  and release it at the end. Thus PREPARE does not 
  accumulate metadata locks (Bug#989/WL#4284).
mysql-test/r/read_only_innodb.result:
  Update test case result (effects of WL#4284/Bug#989).
mysql-test/suite/binlog/r/binlog_row_drop_tbl.result:
  Add a test case result (WL#4284/Bug#989).
mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result:
  Update test case result (effects of WL#4284/Bug#989).
mysql-test/suite/binlog/r/binlog_stm_drop_tbl.result:
  Add a test case result (WL#4284/Bug#989).
mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result:
  Update test case result (effects of WL#4284/Bug#989).
mysql-test/suite/binlog/r/binlog_unsafe.result:
  A side effect of Bug#989 -- slightly different table map ids.
mysql-test/suite/binlog/t/binlog_row_drop_tbl.test:
  Add a test case for WL#4284/Bug#989.
mysql-test/suite/binlog/t/binlog_stm_drop_tbl.test:
  Add a test case for WL#4284/Bug#989.
mysql-test/suite/binlog/t/binlog_stm_row.test:
  Update to the new state name. This
  is actually a follow up to another patch for WL#4284, 
  that changes Locked thread state to Table lock.
mysql-test/suite/ndb/r/ndb_index_ordered.result:
  Remove result for disabled part of the test case.
mysql-test/suite/ndb/t/disabled.def:
  Temporarily disable a test case (Bug#45621).
mysql-test/suite/ndb/t/ndb_index_ordered.test:
  Disable a part of a test case (needs update to
  reflect semantics of Bug#989).
mysql-test/suite/rpl/t/disabled.def:
  Disable tests made meaningless by transactional metadata
  locking.
mysql-test/suite/sys_vars/r/autocommit_func.result:
  Add a commit (Bug#989).
mysql-test/suite/sys_vars/t/autocommit_func.test:
  Add a commit (Bug#989).
mysql-test/t/flush_block_commit.test:
  Fix test case to reflect the fact that transactions now hold
  metadata locks for the duration of a transaction.
mysql-test/t/flush_block_commit_notembedded.test:
  Fix test case to reflect the fact that transactions now hold
  metadata locks for the duration of a transaction.
  Add a test case for transaction-scope locks and the global
  read lock (Bug#989/WL#4284).
mysql-test/t/innodb.test:
  Fix test case to reflect the fact that transactions now hold
  metadata locks for the duration of a transaction
  (effects of Bug#989/WL#4284).
mysql-test/t/lock.test:
  Add a test case for Bug#989/WL#4284.
mysql-test/t/not_embedded_server.test:
  Add a test case for Bug#989/WL#4284.
mysql-test/t/partition_innodb_semi_consistent.test:
  Replace TRUNCATE with DELETE, to not issue
  an implicit commit of a transaction, and not depend
  on metadata locks.
mysql-test/t/partition_sync.test:
  Temporarily disable the test case for Bug#43867,
  which needs a fix to be backported from 6.0.
mysql-test/t/ps.test:
  Add a test case for semantics of PREPARE and transaction-scope
  locks: metadata locks on tables used in PREPARE are enclosed into a
  temporary savepoint, taken at the beginning of PREPARE,
  and released at the end. Thus PREPARE does not effect
  what locks a transaction owns.
mysql-test/t/read_only_innodb.test:
  Fix test case to reflect the fact that transactions now hold
  metadata locks for the duration of a transaction 
  (Bug#989/WL#4284).
  
  Wait for the read_only statement to actually flush tables before
  sending other concurrent statements that depend on its state.
mysql-test/t/xa.test:
  Fix test case to reflect the fact that transactions now hold
  metadata locks for the duration of a transaction 
  (Bug#989/WL#4284).
sql/ha_ndbcluster_binlog.cc:
  Backport bits of changes of ha_ndbcluster_binlog.cc
  from 6.0, to fix the failing binlog test suite with
  WL#4284. WL#4284 implementation does not work
  with 5.1 implementation of ndbcluster binlog index.
sql/log_event.cc:
  Release metadata locks after issuing a commit.
sql/mdl.cc:
  Style changes (WL#4284).
sql/mysql_priv.h:
  Rename parameter to match the name used in the definition (WL#4284).
sql/rpl_injector.cc:
  Release metadata locks on commit (WL#4284).
sql/rpl_rli.cc:
  Remove assert made meaningless, metadata locks are released
  at the end of the transaction.
sql/set_var.cc:
  Close tables and release locks if autocommit mode is set.
sql/slave.cc:
  Release metadata locks after a rollback.
sql/sql_acl.cc:
  Don't implicitly unlock locked tables. Issue a implicit commit
  at the end and unlock tables.
sql/sql_base.cc:
  Defer the release of metadata locks when closing tables
  if not required to.
  Issue a deadlock error if the locking protocol requires
  that a transaction re-acquire its locks.
  
  Release metadata locks when closing tables for reopen.
sql/sql_class.cc:
  Release metadata locks if the thread is killed.
sql/sql_parse.cc:
  Release metadata locks after implicitly committing a active
  transaction, or after explicit commits or rollbacks.
sql/sql_plugin.cc:
  
  Allocate MDL request on the stack as the use of the table
  is contained within the function. It will be removed from
  the context once close_thread_tables is called at the end
  of the function.
sql/sql_prepare.cc:
  The problem is that the prepare phase of the CREATE TABLE
  statement takes a exclusive metadata lock lock and this can
  cause a self-deadlock the thread already holds a shared lock
  on the table being that should be created.
  
  The solution is to make the prepare phase take a shared
  metadata lock when preparing a CREATE TABLE statement. The
  execution of the statement will still acquire a exclusive
  lock, but won't cause any problem as it issues a implicit
  commit.
  
  After some discussions with stakeholders it has been decided that
  metadata locks acquired during a PREPARE statement must be released
  once the statement is prepared even if it is prepared within a multi
  statement transaction.
sql/sql_servers.cc:
  Don't implicitly unlock locked tables. Issue a implicit commit
  at the end and unlock tables.
sql/sql_table.cc:
  Close table and release metadata locks after a admin operation.
sql/table.h:
  The problem is that the prepare phase of the CREATE TABLE
  statement takes a exclusive metadata lock lock and this can
  cause a self-deadlock the thread already holds a shared lock
  on the table being that should be created.
  
  The solution is to make the prepare phase take a shared
  metadata lock when preparing a CREATE TABLE statement. The
  execution of the statement will still acquire a exclusive
  lock, but won't cause any problem as it issues a implicit
  commit.
sql/transaction.cc:
  Release metadata locks after the implicitly committed due
  to a new transaction being started. Also, release metadata
  locks acquired after a savepoint if the transaction is rolled
  back to the save point.
  
  The problem is that in some cases transaction-long metadata locks
  could be released before the transaction was committed. This could
  happen when a active transaction was ended by a "START TRANSACTION"
  or "BEGIN" statement, in which case the metadata locks would be
  released before the actual commit of the active transaction.
  
  The solution is to defer the release of metadata locks to after the
  transaction has been implicitly committed. No test case is provided
  as the effort to provide one is too disproportional to the size of
  the fix.
 2009-12-05 02:02:48 +03:00
Konstantin Osipov
191bb81241 Backport of: 
----------------------------------------------------------
revno: 2617.23.22
committer: Konstantin Osipov <kostja@sun.com>
branch nick: mysql-6.0-runtime
timestamp: Wed 2009-03-04 23:29:16 +0300
message:
  WL#4284, "Transactional DDL locking": fix a Windows compilation warning.
 2009-12-04 02:57:01 +03:00
Konstantin Osipov
a9013f8fba Backport of: 
----------------------------------------------------------
revno: 2617.23.20
committer: Konstantin Osipov <kostja@sun.com>
branch nick: mysql-6.0-runtime
timestamp: Wed 2009-03-04 16:31:31 +0300
message:
  WL#4284 "Transactional DDL locking"
  Review comments: "Objectify" the MDL API.
  MDL_request and MDL_context still need manual construction and
  destruction, since they are used in environment that is averse
  to constructors/destructors.


sql/mdl.cc:
  Improve comments.
  Add asserts to backup()/restore_from_backup()/merge() methods.
  Fix an order bug in the error path of mdl_acquire_exclusive_locks():
  we used to first free a ticket object, and only then exclude
  it from the list of tickets.
 2009-12-04 02:52:05 +03:00
Konstantin Osipov
f477e66ec5 Backport of: 
------------------------------------------------------------
revno: 2617.23.18
committer: Davi Arnaut <Davi.Arnaut@Sun.COM>
branch nick: 4284-6.0
timestamp: Mon 2009-03-02 18:18:26 -0300
message:
Bug#989: If DROP TABLE while there's an active transaction, wrong binlog order
WL#4284: Transactional DDL locking

This is a prerequisite patch:

These changes are intended to split lock requests from granted
locks and to allow the memory and lifetime of granted locks to
be managed within the MDL subsystem. Furthermore, tickets can
now be shared and therefore are used to satisfy multiple lock
requests, but only shared locks can be recursive.

The problem is that the MDL subsystem morphs lock requests into
granted locks locks but does not manage the memory and lifetime
of lock requests, and hence, does not manage the memory of
granted locks either. This can be problematic because it puts the
burden of tracking references on the users of the subsystem and
it can't be easily done in transactional contexts where the locks
have to be kept around for the duration of a transaction.

Another issue is that recursive locks (when the context trying to
acquire a lock already holds a lock on the same object) requires
that each time the lock is granted, a unique lock request/granted
lock structure structure must be kept around until the lock is
released. This can lead to memory leaks in transactional contexts
as locks taken during the transaction should only be released at
the end of the transaction. This also leads to unnecessary wake
ups (broadcasts) in the MDL subsystem if the context still holds
a equivalent of the lock being released.

These issues are exacerbated due to the fact that WL#4284 low-level
design says that the implementation should "2) Store metadata locks
in transaction memory root, rather than statement memory root" but
this is not possible because a memory root, as implemented in mysys,
requires all objects allocated from it to be freed all at once.

This patch combines review input and significant code contributions
from Konstantin Osipov (kostja) and Dmitri Lenev (dlenev).


mysql-test/r/mdl_sync.result:
  Add test case result.
mysql-test/t/mdl_sync.test:
  Add test case for shared lock upgrade case.
sql/event_db_repository.cc:
  Rename mdl_alloc_lock to mdl_request_alloc.
sql/ha_ndbcluster_binlog.cc:
  Use new function names to initialize MDL lock requests.
sql/lock.cc:
  Rename MDL functions.
sql/log_event.cc:
  The MDL request now holds the table and database name data (MDL_KEY).
sql/mdl.cc:
  Move the MDL key to the MDL_LOCK structure in order to make the
  object suitable for allocation from a fixed-size allocator. This
  allows the simplification of the lists in the MDL_LOCK object,
  which now are just two, one for granted tickets and other for
  waiting (upgraders) tickets.
  
  Recursive requests for a shared lock on the same object can now
  be granted using the same lock ticket. This schema is only used
  for shared locks because that the only case that matters. This
  is used to avoid waste of resources in case a context (connection)
  already holds a shared lock on a object.
sql/mdl.h:
  Introduce a metadata lock object key which is used  to uniquely
  identify lock objects.
  
  Separate the structure used to represent pending lock requests
  from the structure used to represent granted metadata locks.
  
  Rename functions used to manipulate locks requests in order to
  have a more consistent function naming schema.
sql/sp_head.cc:
  Rename mdl_alloc_lock to mdl_request_alloc.
sql/sql_acl.cc:
  Rename alloc_mdl_locks to alloc_mdl_requests.
sql/sql_base.cc:
  Various changes to accommodate that lock requests are separated
  from lock tickets (granted locks).
sql/sql_class.h:
  Last acquired lock before the savepoint was set.
sql/sql_delete.cc:
  Various changes to accommodate that lock requests are separated
  from lock tickets (granted locks).
sql/sql_handler.cc:
  Various changes to accommodate that lock requests are separated
  from lock tickets (granted locks).
sql/sql_insert.cc:
  Rename alloc_mdl_locks to alloc_mdl_requests.
sql/sql_parse.cc:
  Rename alloc_mdl_locks to alloc_mdl_requests.
sql/sql_plist.h:
  Typedef for iterator type.
sql/sql_plugin.cc:
  Rename alloc_mdl_locks to alloc_mdl_requests.
sql/sql_servers.cc:
  Rename alloc_mdl_locks to alloc_mdl_requests.
sql/sql_show.cc:
  Various changes to accommodate that lock requests are separated
  from lock tickets (granted locks).
sql/sql_table.cc:
  Various changes to accommodate that lock requests are separated
  from lock tickets (granted locks).
sql/sql_trigger.cc:
  Save reference to the lock ticket so it can be downgraded later.
sql/sql_udf.cc:
  Rename alloc_mdl_locks to alloc_mdl_requests.
sql/table.cc:
  Rename mdl_alloc_lock to mdl_request_alloc.
sql/table.h:
  Separate MDL lock requests from lock tickets (granted locks).
storage/myisammrg/ha_myisammrg.cc:
  Rename alloc_mdl_locks to alloc_mdl_requests.
 2009-12-04 02:29:40 +03:00