Analysis:
Single-row subqueries are not considered expensive and are
evaluated both during EXPLAIN in to detect errors like
"Subquery returns more than 1 row", and during optimization to
perform constant optimization.
The cause for the failed ASSERT is in JOIN::join_free, where we set
bool full= (!select_lex->uncacheable && !thd->lex->describe);
Thus for EXPLAIN statements full == FALSE, and as a result the call to
JOIN::cleanup doesn't call JOIN_TAB::cleanup which should have
called table->disable_keyread().
Solution:
Consider all kinds of subquery predicates as expensive.
Merge 5.3-mwl89 into 5.3 main.
There is one remaining test failure in this merge:
innodb_mysql_lock2. All other tests have been checked to
deliver the same results/explains as 5.3-mwl89, including
the few remaining wrong results.
Phase 3: Implementation of re-optimization of subqueries with injected predicates
and cost comparison between Materialization and IN->EXISTS strategies.
The commit contains the following known problems:
- The implementation of EXPLAIN has not been re-engineered to reflect the
changes in subquery optimization. EXPLAIN for subqueries is called during
the execute phase, which results in different code paths during JOIN::optimize
and thus in differing EXPLAIN messages for constant/system tables.
- There are some valgrind warnings that need investigation
- Several EXPLAINs with minor differences need to be reconsidered after fixing
the EXPLAIN problem above.
This patch also adds one extra optimizer_switch: 'in_to_exists' for complete
manual control of the subquery execution strategies.
Fix some bugs where we stored values other than 0 or 1 in my_bool
Fixed some compiler warnings
client/mysql.cc:
Changed interrupted_query from my_bool to int, as we stored 2 in it.
client/mysqladmin.cc:
Changed return variable type to same type as function value type
client/mysqltest.cc:
Changed 'found' to int as we store other values than 0 or 1 into it
Changed type for parameter of set_reconnect() to match usage.
extra/libevent/evbuffer.c:
Added __attribute__((unused))
extra/libevent/event.c:
Added __attribute__((unused))
extra/libevent/signal.c:
Added __attribute__((unused))
sql/event_data_objects.h:
my_bool -> bool
sql/event_db_repository.cc:
my_bool -> bool
sql/event_db_repository.h:
my_bool -> bool
sql/event_parse_data.h:
my_bool -> bool
sql/events.cc:
my_bool -> bool
sql/events.h:
my_bool -> bool
sql/field.cc:
my_bool -> bool
sql/field.h:
my_bool -> bool
sql/hash_filo.h:
my_bool -> bool
sql/item.cc:
my_bool -> bool
sql/item.h:
my_bool -> bool
sql/item_cmpfunc.h:
my_bool -> bool
Changed result_for_null_param from my_bool to int as we stored -1 in it.
sql/item_func.cc:
my_bool -> bool
Modified udf wrapper functions so that the UDF functions would continue to use my_bool. (To keep compatibility with UDF:s)
sql/item_func.h:
my_bool -> bool
sql/item_subselect.h:
my_bool -> bool
sql/item_sum.cc:
Modified udf wrapper functions so that the UDF functions would continue to use my_bool. (To keep compatibility with UDF:s)
sql/parse_file.h:
my_bool -> bool
sql/rpl_mi.h:
my_bool -> bool
sql/sp_rcontext.h:
my_bool -> bool
sql/sql_analyse.h:
my_bool -> bool
sql/sql_base.cc:
Change some assignments so that we don't initialize bool variables with int's.
sql/sql_bitmap.h:
my_bool -> bool
sql/sql_cache.cc:
my_bool -> bool
sql/sql_cache.h:
my_bool -> bool
sql/sql_class.h:
my_bool -> bool
sql/sql_insert.cc:
Change some assignments so that we don't initialize bool variables with int's.
sql/sql_prepare.cc:
my_bool -> bool
sql/table.h:
my_bool -> bool
storage/maria/ma_check.c:
Removed duplicate assignment
strings/decimal.c:
Fixed wrong variable usage.
Don't do complex arithmetic on bool when simple works.
Step2 in the separation of the creation of IN->EXISTS equi-join conditions from
their injection. The goal of this separation is to make it possible that the
IN->EXISTS conditions can be used for cost estimation without actually modifying
the subquery.
This patch separates row_value_in_to_exists_transformer() into two methods:
- create_row_value_in_to_exists_cond(), and
- inject_row_value_in_to_exists_cond()
The patch performs minimal refactoring of the code so that it is easier to solve
problems resulting from the separation. There is a lot to be simplified in this
code, but this will be done separately.
Step1 in the separation of the creation of IN->EXISTS equi-join conditions from
their injection. The goal of this separation is to make it possible that the
IN->EXISTS conditions can be used for cost estimation without actually modifying
the subquery.
This patch separates single_value_in_to_exists_transformer() into two methods:
- create_single_value_in_to_exists_cond(), and
- inject_single_value_in_to_exists_cond()
The patch performs minimal refactoring of the code so that it is easier to solve
problems resulting from the separation. There is a lot to be simplified in this
code, but this will be done separately.
1. Changed the lazy optimization for subqueries that can be
materialized into bottom-up optimization during the optimization of
the main query.
The main change is implemented by the method
Item_in_subselect::setup_engine.
All other changes were required to correct problems resulting from
changing the order of optimization. Most of these problems followed
the same pattern - there are some shared structures between a
subquery and its parent query. Depending on which one is optimized
first (parent or child query), these shared strucutres may get
different values, thus resulting in an inconsistent query plan.
2. Changed the code-generation for subquery materialization to be
performed in runtime memory for each (re)execution, instead of in
statement memory (once per prepared statement).
- Item_in_subselect::setup_engine() no longer creates materialization
related objects in statement memory.
- Merged subselect_hash_sj_engine::init_permanent and
subselect_hash_sj_engine::init_runtime into
subselect_hash_sj_engine::init, which is called for each
(re)execution.
- Fixed deletion of the temp table accordingly.
mysql-test/r/subselect_mat.result:
Adjusted changed EXPLAIN because of earlier optimization of subqueries.
libmysqld/Makefile.am:
The new file added.
mysql-test/r/index_merge_myisam.result:
subquery_cache optimization option added.
mysql-test/r/myisam_mrr.result:
subquery_cache optimization option added.
mysql-test/r/subquery_cache.result:
The subquery cache tests added.
mysql-test/r/subselect3.result:
Subquery cache switched off to avoid changing read statistics.
mysql-test/r/subselect3_jcl6.result:
Subquery cache switched off to avoid changing read statistics.
mysql-test/r/subselect_no_mat.result:
subquery_cache optimization option added.
mysql-test/r/subselect_no_opts.result:
subquery_cache optimization option added.
mysql-test/r/subselect_no_semijoin.result:
subquery_cache optimization option added.
mysql-test/r/subselect_sj.result:
subquery_cache optimization option added.
mysql-test/r/subselect_sj_jcl6.result:
subquery_cache optimization option added.
mysql-test/t/subquery_cache.test:
The subquery cache tests added.
mysql-test/t/subselect3.test:
Subquery cache switched off to avoid changing read statistics.
sql/CMakeLists.txt:
The new file added.
sql/Makefile.am:
The new files added.
sql/item.cc:
Expression cache item (Item_cache_wrapper) added.
Item_ref and Item_field fixed for correct usage of result field and fast resolwing in SP.
sql/item.h:
Expression cache item (Item_cache_wrapper) added.
Item_ref and Item_field fixed for correct usage of result field and fast resolwing in SP.
sql/item_cmpfunc.cc:
Subquery cache added.
sql/item_cmpfunc.h:
Subquery cache added.
sql/item_subselect.cc:
Subquery cache added.
sql/item_subselect.h:
Subquery cache added.
sql/item_sum.cc:
Registration of subquery parameters added.
sql/mysql_priv.h:
subquery_cache optimization option added.
sql/mysqld.cc:
subquery_cache optimization option added.
sql/opt_range.cc:
Fix due to subquery cache.
sql/opt_subselect.cc:
Parameters of the function cahnged.
sql/procedure.h:
.h file guard added.
sql/sql_base.cc:
Registration of subquery parameters added.
sql/sql_class.cc:
Option to allow add indeces to temporary table.
sql/sql_class.h:
Item iterators added.
Option to allow add indeces to temporary table.
sql/sql_expression_cache.cc:
Expression cache for caching subqueries added.
sql/sql_expression_cache.h:
Expression cache for caching subqueries added.
sql/sql_lex.cc:
Registration of subquery parameters added.
sql/sql_lex.h:
Registration of subqueries and subquery parameters added.
sql/sql_select.cc:
Subquery cache added.
sql/sql_select.h:
Subquery cache added.
sql/sql_union.cc:
A new parameter to the function added.
sql/sql_update.cc:
A new parameter to the function added.
sql/table.cc:
Procedures to manage temporarty tables index added.
sql/table.h:
Procedures to manage temporarty tables index added.
storage/maria/ha_maria.cc:
Fix of handler to allow destoy a table in case of error during the table creation.
storage/maria/ha_maria.h:
.h file guard added.
storage/myisam/ha_myisam.cc:
Fix of handler to allow destoy a table in case of error during the table creation.
Fixed some bugs introduced in 5.1.47
Disabled some tests until we have merged with latest Xtradb
configure.in:
Added testing if valgrind/memcheck.h exists
storage/pbxt/src/ha_pbxt.cc:
LOCK_plugin is not anymore locked in init
We should disable const subselect item evaluation because
subselect transformation does not happen in view_prepare_mode
and thus val_...() methods can not be called.
mysql-test/r/ctype_ucs.result:
test case
mysql-test/r/view.result:
test case
mysql-test/t/ctype_ucs.test:
test case
mysql-test/t/view.test:
test case
sql/item.cc:
disabled const subselect item evaluation in
view prepare mode.
sql/item_subselect.cc:
added Item_subselect::safe_charset_converter which
prevents const item evaluation in view prepare mode.
sql/item_subselect.h:
added Item_subselect::safe_charset_converter which
prevents const item evaluation in view prepare mode.
This patch:
- Moves all definitions from the mysql_priv.h file into
header files for the component where the variable is
defined
- Creates header files if the component lacks one
- Eliminates all include directives from mysql_priv.h
- Eliminates all circular include cycles
- Rename time.cc to sql_time.cc
- Rename mysql_priv.h to sql_priv.h
- Add Item_in_subselect::get_identifier() that returns subquery's id
- Change select_describe() to produce output in new format
- Update test results (checked)
Bug#46949: memory leak with failed alter table to create partitions based on extract()
Bug#51830: Incorrect partition pruning on range partition (regression)
Fixed valgrind failure in select_describe(), read of uninitialized
Item_subselect::eliminated.
PBXT test file updates to reflect changes done in MySQL.
mysql-test/suite/pbxt/r/partition_error.result:
Result file update following MySQL 5.1.44 changes.
mysql-test/suite/pbxt/r/partition_pruning.result:
Result file update following MySQL 5.1.44 changes.
mysql-test/suite/pbxt/t/partition_error.test:
Test file update following MySQL 5.1.44 changes.
sql/item_subselect.cc:
Fixed valgrind failure in select_describe(), read of uninitialized
Item_subselect::eliminated:
- it turns out we can call select_describe() without having fixed
subquery items for child subselects. These are not the kind of subqueries
that we could eliminate, so the fix is to ensure that
item_subselect->eliminated==FALSE even before fix_fields is called.
Also added code to reset item_subselect->eliminated back to FALSE in
Item::reset() call.
sql/item_subselect.h:
Fixed valgrind failure in select_describe(), read of uninitialized
Item_subselect::eliminated:
- it turns out we can call select_describe() without having fixed
subquery items for child subselects. These are not the kind of subqueries
that we could eliminate, so the fix is to ensure that
item_subselect->eliminated==FALSE even before fix_fields is called.
Also added code to reset item_subselect->eliminated back to FALSE in
Item::reset() call.
sql/sql_partition.cc:
Fix Bug#51830: Revert part of the patch for Bug#49742, which caused the regression.
sql/table.cc:
Fix Bug#46949: memory leak in failed ALTER TABLE with partitioning.
* Implemented a second partial matching strategy via table scan.
This strategy is a fallback when there is no memory for rowid merging.
* Refactored the selection and creation of partial matching strategies,
so that the choice of strategy is encapsulated in a separate method
choose_partial_match_strategy().
* Refactored the representation of partial match strategies so that:
- each strategy is represented by a polymorphic class, and
- the base class for all partial match strategies contains common
execution code.
* Added an estimate of the memory needed for the rowid merge strategy,
and the system variable "rowid_merge_buff_size" to control the maximum
memory to be used by the rowid merge algorithm.
* Added two optimizer_switch system variables to control the choice of
partial match strategy:
"partial_match_rowid_merge", "partial_match_table_scan".
* Fixed multiple problems with deallocation of resources by the partial
match strategies.
sql/mysql_priv.h:
* Added two optimizer_switch system variables to control the choice of
partial match strategy:
"partial_match_rowid_merge", "partial_match_table_scan".
sql/mysqld.cc:
* Added two optimizer_switch system variables to control the choice of
partial match strategy:
"partial_match_rowid_merge", "partial_match_table_scan".
* Added a system variable "rowid_merge_buff_size" to control the maximum
memory to be used by the rowid merge algorithm.
sql/set_var.cc:
* Added a system variable "rowid_merge_buff_size" to control the maximum
memory to be used by the rowid merge algorithm.
sql/sql_class.h:
* Added a system variable "rowid_merge_buff_size" to control the maximum
memory to be used by the rowid merge algorithm.
support-files/build-tags:
Newer versions of BZR require the recursive flag in order to list all files.
This patch mainly adds sorting of all indexes for partial matching
according to their NULL selectivity. The patch also fixes a related bug
in subselect_rowid_merge_engine::test_null_row() where the wrong matched
indexes were skipped.
In addition the patch:
- adds few ::print() methods,
- renames few variables that had similar names but different purpose.
This patch implements correct NULL semantics for materialized subquery execution.
The implementation has the following properties and main limitations:
- It passes all query result tests, but fails a number of EXPLAIN tests because of
changed plans.
- The EXPLAIN output for partial matching is not decided yet.
- It works only when all necessary indexes fit into main memory. Notice that these
are not the general B-tree/Hash indexes, but instead much more compact ones,
therefore this limitation may not be a problem in many practical cases.
- It doesn't contain specialized tests.
- In several places the implementation uses methods that are modified copies of
other similar methods. These cases need to be refactored to avoid code duplication.
- Add a test if the predicate is top-level just before deciding on partial matching.
If it is top-level, use a more efficient exec method (index lookup).
- Add sorting of indexes according to their selectivity. The code is almost there.
- Needs more comments, and to sync existing ones with the implementation.
sql/item_cmpfunc.h:
Expose the Arg_comparator of a comparison predicate. This makes it possible to
directly get the comparison result {-1,0,1}, which is not possible through the
val_XXX() methods which "fold" such results into a boolean.
sql/item_subselect.cc:
The core of the implementation of MWL#68.
sql/item_subselect.h:
The core of the implementation of MWL#68.
sql/opt_subselect.cc:
Removed the limitation for materialized subquery execution that it is applicable only
for top-level predicates.
sql/sql_class.cc:
New class select_materialize_with_stats that collects data statistics about
the data being inserted into the target table.
sql/sql_class.h:
New class select_materialize_with_stats that collects data statistics about
the data being inserted into the target table.
sql/sql_select.cc:
- more complete initialization of the TABLE object of a temp table.
- call setup_subquery_materialization at one more exit point.
- Marked a couple of tests with --big
- Fixed xtradb/handler/ha_innodb.cc to call explain_filename()
storage/xtradb/handler/ha_innodb.cc:
Call explain_filename() to get proper names for partitioned tables
MySQL manual describes values of the YEAR(2) field type as follows:
values 00 - 69 mean 2000 - 2069 years and values 70 - 99 mean 1970 - 1999
years. MIN/MAX and comparison functions was comparing them as int values
thus producing wrong result.
Now the Arg_comparator class is extended with compare_year function which
performs correct comparison of the YEAR type.
The Item_sum_hybrid class now uses Item_cache and Arg_comparator objects to
correctly calculate its value.
To allow Arg_comparator to use func_name() function for Item_func and Item_sum
objects the func_name declaration is moved to the Item_result_field class.
A helper function is_owner_equal_func is added to the Arg_comparator class.
It checks whether the Arg_comparator object owner is the <=> function or not.
A helper function setup is added to the Item_sum_hybrid class. It sets up
cache item and comparator.
mysql-test/r/func_group.result:
Added a test case for the bug#43668.
mysql-test/t/func_group.test:
Added a test case for the bug#43668.
sql/item.cc:
Bug#43668: Wrong comparison and MIN/MAX for YEAR(2)
Now Item_cache_int returns the type of cached item.
sql/item.h:
Bug#43668: Wrong comparison and MIN/MAX for YEAR(2)
To allow Arg_comparator to use func_name() function for Item_func and Item_sum
objects the func_name declaration is moved to the Item_result_field class.
sql/item_cmpfunc.cc:
Bug#43668: Wrong comparison and MIN/MAX for YEAR(2)
The Arg_comparator class is extended with compare_year function which
performs correct comparison of the YEAR type.
sql/item_cmpfunc.h:
Bug#43668: Wrong comparison and MIN/MAX for YEAR(2)
The year_as_datetime variable is added to the Arg_comparator class.
It's set to TRUE when YEAR value should be converted to the
YYYY-00-00 00:00:00 format for correct YEAR-DATETIME comparison.
sql/item_geofunc.cc:
Bug#43668: Wrong comparison and MIN/MAX for YEAR(2)
Item_func_spatial_rel::val_int chenged to use Arg_comparator's string
buffers.
sql/item_subselect.h:
Bug#43668: Wrong comparison and MIN/MAX for YEAR(2)
Added an implementation of the virtual func_name function.
sql/item_sum.cc:
Bug#43668: Wrong comparison and MIN/MAX for YEAR(2)
The Item_sum_hybrid class now uses Item_cache and Arg_comparator objects to
correctly calculate its value.
A helper function setup is added to the Item_sum_hybrid class. It sets up
cache item and comparator.
sql/item_sum.h:
Bug#43668: Wrong comparison and MIN/MAX for YEAR(2)
The Item_sum_hybrid class now uses Item_cache and Arg_comparator objects to
correctly calculate its value.
Added an implementation of the virtual func_name function.
- Make elimination check to be able detect cases like t.primary_key_col1=othertbl.col AND t.primary_key_col2=func(t.primary_key_col1).
These are needed to handle e.g. the case of func() being a correlated subquery that selects the latest value.
- If we've removed a condition with subquery predicate, EXPLAIN [EXTENDED] won't show the subquery anymore
sql/item.cc:
MWL#17: Table elimination
- Add tem_field::check_column_usage_processor(). it allows to check which key parts a condition depends on.
sql/item.h:
MWL#17: Table elimination
- Add tem_field::check_column_usage_processor(). it allows to check which key parts a condition depends on.
sql/item_subselect.cc:
MWL#17: Table elimination
- Item_subselect got 'eliminated' attribute. It is used only to determine if the subselect should be printed by EXPLAIN.
- Item_subselect got List<Item> refers_to - a list of item in the current select that are referred to from within the subselect.
- Added Item_*::check_column_usage_processor(). it allows to check which key parts a condition depends on.
- Added a comment about possible problem in Item_subselect::walk
sql/item_subselect.h:
MWL#17: Table elimination
- Item_subselect got 'eliminated' attribute. It is used only to determine if the subselect should be printed by EXPLAIN.
- Item_subselect got List<Item> refers_to - a list of item in the current select that are referred to from within the subselect.
- Added Item_*::check_column_usage_processor(). it allows to check which key parts a condition depends on.
sql/item_sum.cc:
MWL#17: Table elimination
sql/sql_lex.cc:
MWL#17: Table elimination
sql/sql_lex.h:
MWL#17: Table elimination
sql/sql_select.h:
MWL#17: Table elimination
between 5.0 and 5.1.
The problem was that in the patch for Bug#11986 it was decided
to store original query in UTF8 encoding for the INFORMATION_SCHEMA.
This approach however turned out to be quite difficult to implement
properly. The main problem is to preserve the same IS-output after
dump/restore.
So, the fix is to rollback to the previous functionality, but also
to fix it to support multi-character-set-queries properly. The idea
is to generate INFORMATION_SCHEMA-query from the item-tree after
parsing view declaration. The IS-query should:
- be completely in UTF8;
- not contain character set introducers.
For more information, see WL4052.
mysql-test/include/ddl_i18n.check_views.inc:
Add a test case for Bug#30217.
mysql-test/r/ddl_i18n_koi8r.result:
Update result file.
mysql-test/r/ddl_i18n_utf8.result:
Update result file.
mysql-test/r/information_schema.result:
Update result file.
mysql-test/r/information_schema_db.result:
Update result file.
mysql-test/r/mysqldump.result:
Update result file.
mysql-test/r/show_check.result:
Update result file.
mysql-test/t/ddl_i18n_koi8r.test:
Add a test case for Bug#30217.
mysql-test/t/ddl_i18n_utf8.test:
Add a test case for Bug#30217.
mysql-test/t/mysqldump.test:
Add a test case for Bug#30217.
sql/ha_ndbcluster.cc:
Add a parameter to print().
sql/item.cc:
1. Add a parameter to print().
2. Item_string::print():
- Do not append character set introducer to the text literal
if we're building a query for INFORMATION_SCHEMA;
- Convert text literal to UTF8 if we're building a query
for INFORMATION_SCHEMA.
sql/item.h:
Add a parameter to print().
sql/item_cmpfunc.cc:
Add a parameter to print().
sql/item_cmpfunc.h:
Add a parameter to print().
sql/item_func.cc:
Add a parameter to print().
sql/item_func.h:
Add a parameter to print().
sql/item_geofunc.h:
Add a parameter to print().
sql/item_row.cc:
Add a parameter to print().
sql/item_row.h:
Add a parameter to print().
sql/item_strfunc.cc:
Add a parameter to print().
sql/item_strfunc.h:
Add a parameter to print().
sql/item_subselect.cc:
Add a parameter to print().
sql/item_subselect.h:
Add a parameter to print().
sql/item_sum.cc:
Add a parameter to print().
sql/item_sum.h:
Add a parameter to print().
sql/item_timefunc.cc:
Add a parameter to print().
sql/item_timefunc.h:
Add a parameter to print().
sql/mysql_priv.h:
Add a parameter to print().
sql/sp_head.cc:
Add a parameter to print().
sql/sql_lex.cc:
Add a parameter to print().
sql/sql_lex.h:
Add a parameter to print().
sql/sql_parse.cc:
Add a parameter to print().
sql/sql_select.cc:
Add a parameter to print().
sql/sql_show.cc:
Add a parameter to print().
sql/sql_test.cc:
Add a parameter to print().
sql/sql_view.cc:
Build INFORMATION_SCHEMA query from Item-tree.
sql/sql_yacc.yy:
Build INFORMATION_SCHEMA query from Item-tree.
sql/table.h:
Add a parameter to print().
into magare.gmz:/home/kgeorge/mysql/work/B31884-5.1-opt
sql/item_subselect.h:
Auto merged
mysql-test/r/subselect.result:
merged the fix for bug 31884 to 5.1-opt
mysql-test/t/subselect.test:
merged the fix for bug 31884 to 5.1-opt
