The problem was that optimizer removes some outer references (it they are
constant for example) and the list of outer items built during prepare phase is
not actual during execution phase when we need it as the cache parameters.
First solution was use pointer on pointer on outer reference Item and
initialize temporary table on demand. This solved most problem except case
when optimiser also reduce Item which contains outer references ('OR' in
this bug test suite).
The solution is to build the list of outer reference items on execution
phase (after optimization) on demand (just before temporary table creation)
by walking Item tree and finding outer references among Item_ident
(Item_field/Item_ref) and Item_sum items.
Removed depends_on list (because it is not neede any mnore for the cache, in the place where it was used it replaced with upper_refs).
Added processor (collect_outer_ref_processor) and get_cache_parameters() methods to collect outer references (or other expression parameters in future).
mysql-test/r/subselect_cache.result:
A new test added.
mysql-test/r/subselect_scache.result:
Changes in creating the cache and its paremeters order or adding arguments of aggregate function (which is a parameter also, but this has no influence on the result).
mysql-test/t/subselect_cache.test:
Added a new test.
sql/item.cc:
depends_on removed.
Added processor (collect_outer_ref_processor) and get_cache_parameters() methods to collect outer references.
Item_cache_wrapper collect parameters befor initialization of its cache.
sql/item.h:
depends_on removed.
Added processor (collect_outer_ref_processor) and get_cache_parameters() methods to collect outer references.
sql/item_cmpfunc.cc:
depends_on removed.
Added processor (collect_outer_ref_processor) to collect outer references.
sql/item_cmpfunc.h:
Added processor (collect_outer_ref_processor) to collect outer references.
sql/item_subselect.cc:
depends_on removed.
Added processor get_cache_parameters() method to collect outer references.
sql/item_subselect.h:
depends_on removed.
Added processor get_cache_parameters() method to collect outer references.
sql/item_sum.cc:
Added processor (collect_outer_ref_processor) method to collect outer references.
sql/item_sum.h:
Added processor (collect_outer_ref_processor) and get_cache_parameters() methods to collect outer references.
sql/opt_range.cc:
depends_on removed.
sql/sql_base.cc:
depends_on removed.
sql/sql_class.h:
New iterator added.
sql/sql_expression_cache.cc:
Build of list of items resolved in outer query done just before creating expression cache on the first execution of the subquery which removes influence of optimizer removing items (all optimization already done).
sql/sql_expression_cache.h:
Build of list of items resolved in outer query done just before creating expression cache on the first execution of the subquery which removes influence of optimizer removing items (all optimization already done).
sql/sql_lex.cc:
depends_on removed.
sql/sql_lex.h:
depends_on removed.
sql/sql_list.h:
Added add_unique method to add only unique elements to the list.
sql/sql_select.cc:
Support of new Item list added.
sql/sql_select.h:
Support of new Item list added.
Analysis:
Both the wrong result and the valgrind warning were a result
of incomplete cleanup of the MIN/MAX subquery rewrite. At the
first execution of the query, the non-aggregate subquery is
transformed into an aggregate MIN/MAX subquery. During the
fix_fields phase of the MIN/MAX function, it sets the property
st_select_lex::with_sum_func to true.
The second execution of the query finds this flag to be ON.
When optimization reaches the same MIN/MAX subquery
transformation, it tests if the subquery is an aggregate or not.
Since select_lex->with_sum_func == true from the previous
execution, the transformation executes the second branch that
handles aggregate subqueries. This substitutes the subquery
Item into a Item_maxmin_subselect. At the same time elsewhere
it is assumed that the subquery Item is of type
Item_allany_subselect. Ultimately this results in casting the
actual object to the wrong class, and calling the wrong
any_value() method from empty_underlying_subquery().
Solution:
Cleanup the st_select_lex::with_sum_func property in the case
when the MIN/MAX transformation was performed for a non-aggregate
subquery, so that the transformation can be repeated.
Also:
1. simplified the code of the function mysql_derived_merge_for_insert.
2. moved merge of views/dt for multi-update/delete to the prepare stage.
3. the list of the references to the candidates for semi-join now is
allocated in the statement memory.
Analysis:
This is a bug in MWL#68, where it was incorrectly assumed
that if there is a match in the only non-null key, then
if there is a covering NULL row on all remaining NULL-able
columns there is a partial match. However, this is not the case,
because even if there is such a null-only sub-row, it is not
guaranteed to be part of the matched sub-row. The matched sub-row
and the NULL-only sub-row may be parts of different rows.
In fact there are two cases:
- there is a complete row with only NULL values, and
- all nullable columns contain only NULL values.
These two cases were incorrectly mixed up in the class member
subselect_partial_match_engine::covering_null_row_width.
Solution:
The solution is to:
- split covering_null_row_width into two members:
has_covering_null_row, and has_covering_null_columns, and
- take into account each state during initialization and
execution.
In addition to the bug fix explained below, the patch performs
few renames, and adds some comments to avoid similar problems.
Analysis:
The failed assert was due to a bug in MWL#68, where it was
incorrectly assumed that the size of the bitmap
subselect_rowid_merge_engine::null_only_columns should be
the same as the size of the array of Ordered_keys.
The bitmap null_only_columns contains bits to mark columns
that contain only NULLs. Therefore the indexes of the bits
to be set in null_only_columns are different from the indexes
of the Ordered_keys. If there is a NULL-only column that appears
in a table after the last partial match column with Ordered_key,
this NULL-only column would require setting a bit with index
bigger than the size of the bitmap null_only_columns.
Accessing such a bit caused the failed assert.
Solution:
Upon analysis, it turns out that null_only_columns is not needed
at all, because we are looking for partial matches, and having
such columns guarantees that there is a partial match for any
corresponding outer value.
Therefore the patch removes
subselect_rowid_merge_engine::null_only_columns.
- Added regression test with queries over the WORLD database.
- Discovered and fixed several bugs in the related cost calculation
functionality both in the semijoin and non-semijon subquery code.
- Added DBUG printing of the cost variables used to decide between
IN-EXISTS and MATERIALIZATION.
way of processing prepared statements:
- conversion subquery_predicate -> TABLE_LIST is once per-statement
- However, the code must take into account that materialized temptable
is dropped and re-created on each execution (the tricky part is that
at start of n-th EXECUTE we have TABLE_LIST object but not its TABLE object)
- IN-equality is injected into WHERE on every execution.
mysql-test/r/subselect4.result:
Moved test case for LP BUG#718593 into the correct test file subselect_mat_cost_bugs.test.
mysql-test/t/subselect4.test:
Moved test case for LP BUG#718593 into the correct test file subselect_mat_cost_bugs.test.
Resolved all conflicts, bad merges and fixed a few minor bugs in the code.
Commented out the queries from multi_update, view, subselect_sj, func_str,
derived_view, view_grant that failed either with crashes in ps-protocol or
with wrong results.
The failures are clear indications of some bugs in the code and these bugs
are to be fixed.
Analysis:
The subquery is evaluated first during ref-optimization of the outer
query because the subquery is considered constant from the perspective
of the outer query. Thus an attempt is made to evaluate the MAX subquery
and use the new constant to drive an index nested loops join.
During this evaluation the inner-most subquery replaces the JOIN_TAB
with a new one that fetches the data from a temp table.
The function select_describe crashes at the lines:
TABLE_LIST *real_table= table->pos_in_table_list;
item_list.push_back(new Item_string(real_table->alias,
strlen(real_table->alias),
cs));
because 'table' is a temp table, and it has no corresponding table
reference. This 'real_table' is NULL, and real_table->alias results
in a crash.
Solution:
In the spirit of MWL#89 prevent the evaluation of expensive predicates
during optimization. This patch prevents the evaluation of expensive
predicates during ref optimization.
sql/item_subselect.h:
Remove unused class member. Not needed for the fix, but noticed now and removed.
sql/item_subselect.cc:
Cleanup. Comments added.
sql/item_subselect.h:
Cleanup.
sql/mysql_priv.h:
Comments added.
sql/opt_subselect.cc:
The function renamed and turned to method.
Comments added.
sql/opt_subselect.h:
The function turned to method of JOIN.
sql/sql_select.cc:
Comment added. The function turned to method.
sql/sql_select.h:
The function turned to method.
mysql-test/r/explain.result:
fixed results (new item)
mysql-test/r/subselect.result:
fixed results (new item)
mysql-test/r/subselect_no_mat.result:
fixed results (new item)
mysql-test/r/subselect_no_opts.result:
fixed results (new item)
mysql-test/r/subselect_no_semijoin.result:
Fixed results (new item)
mysql-test/suite/pbxt/r/subselect.result:
Fixed results (new item)
mysql-test/t/explain.test:
Fixed results (correct behaviour)
sql/item_cmpfunc.cc:
Pass through for max/min
sql/item_subselect.cc:
moving max/min
sql/item_subselect.h:
moving max/min
sql/mysql_priv.h:
new uncacheble flags added
sql/opt_subselect.cc:
maxmin moved.
sql/opt_subselect.h:
New function for maxmin.
sql/sql_class.h:
debug code
sql/sql_lex.cc:
Fixed flags.
Limit setting fixed.
sql/sql_lex.h:
2 new flags.
sql/sql_select.cc:
Prepare divided on 2 function to be able recollect some info after transformation.
sql/sql_select.h:
Prepare divided on 2 functions.
- Fixed some issues with partitions and connection_string, which also fixed lp:716890 "Pre- and post-recovery crash in Aria"
- Fixed wrong assert in Aria
Now need to merge with latest xtradb before pushing
sql/ha_partition.cc:
Ensure that m_ordered_rec_buffer is not freed before close.
sql/mysqld.cc:
Changed to use opt_stack_trace instead of opt_pstack.
Removed references to pstack
sql/partition_element.h:
Ensure that connect_string is initialized
storage/maria/ma_key_recover.c:
Fixed wrong assert
Analysis:
The assert failed because st_select_lex::print() was called for subqueries
as follows:
Item_subselect::print() ->
subselect_single_select_engine::print() -> st_select_lex::print()
It was Item_subselect::fix_fields() that set the thd by calling set_thd(),
so when this print() was called before fix_fields(), subselect_engine::thd
was NULL.
Solution:
The patch makes all constructors of all subselect_engine classes to take
a THD parameter. The default subselect_single_select_engine engine is created
early during parse time, in the Item_subselect::init call, so we pass the
correct THD object already at this point.
--Bug#52157 various crashes and assertions with multi-table update, stored function
--Bug#54475 improper error handling causes cascading crashing failures in innodb/ndb
--Bug#57703 create view cause Assertion failed: 0, file .\item_subselect.cc, line 846
--Bug#57352 valgrind warnings when creating view
--Recently discovered problem when a nested materialized derived table is used
before being populated and it leads to incorrect result
We have several modes when we should disable subquery evaluation.
The reasons for disabling are different. It could be
uselessness of the evaluation as in case of 'CREATE VIEW'
or 'PREPARE stmt', or we should disable subquery evaluation
if tables are not locked yet as it happens in bug#54475, or
too early evaluation of subqueries can lead to wrong result
as it happened in Bug#19077.
Main problem is that if subquery items are treated as const
they are evaluated in ::fix_fields(), ::fix_length_and_dec()
of the parental items as a lot of these methods have
Item::val_...() calls inside.
We have to make subqueries non-const to prevent unnecessary
subquery evaluation. At the moment we have different methods
for this. Here is a list of these modes:
1. PREPARE stmt;
We use UNCACHEABLE_PREPARE flag.
It is set during parsing in sql_parse.cc, mysql_new_select() for
each SELECT_LEX object and cleared at the end of PREPARE in
sql_prepare.cc, init_stmt_after_parse(). If this flag is set
subquery becomes non-const and evaluation does not happen.
2. CREATE|ALTER VIEW, SHOW CREATE VIEW, I_S tables which
process FRM files
We use LEX::view_prepare_mode field. We set it before
view preparation and check this flag in
::fix_fields(), ::fix_length_and_dec().
Some bugs are fixed using this approach,
some are not(Bug#57352, Bug#57703). The problem here is
that we have a lot of ::fix_fields(), ::fix_length_and_dec()
where we use Item::val_...() calls for const items.
3. Derived tables with subquery = wrong result(Bug19077)
The reason of this bug is too early subquery evaluation.
It was fixed by adding Item::with_subselect field
The check of this field in appropriate places prevents
const item evaluation if the item have subquery.
The fix for Bug19077 fixes only the problem with
convert_constant_item() function and does not cover
other places(::fix_fields(), ::fix_length_and_dec() again)
where subqueries could be evaluated.
Example:
CREATE TABLE t1 (i INT, j BIGINT);
INSERT INTO t1 VALUES (1, 2), (2, 2), (3, 2);
SELECT * FROM (SELECT MIN(i) FROM t1
WHERE j = SUBSTRING('12', (SELECT * FROM (SELECT MIN(j) FROM t1) t2))) t3;
DROP TABLE t1;
4. Derived tables with subquery where subquery
is evaluated before table locking(Bug#54475, Bug#52157)
Suggested solution is following:
-Introduce new field LEX::context_analysis_only with the following
possible flags:
#define CONTEXT_ANALYSIS_ONLY_PREPARE 1
#define CONTEXT_ANALYSIS_ONLY_VIEW 2
#define CONTEXT_ANALYSIS_ONLY_DERIVED 4
-Set/clean these flags when we perform
context analysis operation
-Item_subselect::const_item() returns
result depending on LEX::context_analysis_only.
If context_analysis_only is set then we return
FALSE that means that subquery is non-const.
As all subquery types are wrapped by Item_subselect
it allow as to make subquery non-const when
it's necessary.
mysql-test/r/derived.result:
test case
mysql-test/r/multi_update.result:
test case
mysql-test/r/view.result:
test case
mysql-test/suite/innodb/r/innodb_multi_update.result:
test case
mysql-test/suite/innodb/t/innodb_multi_update.test:
test case
mysql-test/suite/innodb_plugin/r/innodb_multi_update.result:
test case
mysql-test/suite/innodb_plugin/t/innodb_multi_update.test:
test case
mysql-test/t/derived.test:
test case
mysql-test/t/multi_update.test:
test case
mysql-test/t/view.test:
test case
sql/item.cc:
--removed unnecessary code
sql/item_cmpfunc.cc:
--removed unnecessary checks
--THD::is_context_analysis_only() is replaced with LEX::is_ps_or_view_context_analysis()
sql/item_func.cc:
--refactored context analysis checks
sql/item_row.cc:
--removed unnecessary checks
sql/item_subselect.cc:
--removed unnecessary code
--added DBUG_ASSERT into Item_subselect::exec()
which asserts that subquery execution can not happen
if LEX::context_analysis_only is set, i.e. at context
analysis stage.
--Item_subselect::const_item()
Return FALSE if LEX::context_analysis_only is set.
It prevents subquery evaluation in ::fix_fields &
::fix_length_and_dec at context analysis stage.
sql/item_subselect.h:
--removed unnecessary code
sql/mysql_priv.h:
--Added new set of flags.
sql/sql_class.h:
--removed unnecessary code
sql/sql_derived.cc:
--added LEX::context_analysis_only analysis intialization/cleanup
sql/sql_lex.cc:
--init LEX::context_analysis_only field
sql/sql_lex.h:
--New LEX::context_analysis_only field
sql/sql_parse.cc:
--removed unnecessary code
sql/sql_prepare.cc:
--removed unnecessary code
--added LEX::context_analysis_only analysis intialization/cleanup
sql/sql_select.cc:
--refactored context analysis checks
sql/sql_show.cc:
--added LEX::context_analysis_only analysis intialization/cleanup
sql/sql_view.cc:
--added LEX::context_analysis_only analysis intialization/cleanup
Changed rows_read and rows_sent status variables to be longlong (to avoid compiler warnings)
sql/item_func.cc:
Fixed wrong usage of alias
sql/item_subselect.cc:
Changed buffer size to ulonglong to be able detect buffers bigger than size_t
sql/item_subselect.h:
Changed buffer size to ulonglong to be able detect buffers bigger than size_t
sql/multi_range_read.cc:
Fixed compiler warning by using correct type for function argument
sql/mysqld.cc:
Changed rows_read and rows_sent status variables to be longlong
sql/opt_subselect.h:
Fixed compiler warning by using correct type for function argument
sql/sql_class.cc:
Changed rows_read and rows_sent status variables to be longlong
sql/sql_class.h:
Changed rows_read and rows_sent status variables to be longlong
Changed max_nulls_in_row to uint as this is number of columns in row.
This fixed some compiler warnings.
sql/sql_select.cc:
Added casts to avoid compiler warnings
storage/heap/ha_heap.cc:
Initilize different types separate
storage/oqgraph/ha_oqgraph.cc:
Fixed argument to store(longlong) to avoid compiler warnings
Analysis:
Single-row subqueries are not considered expensive and are
evaluated both during EXPLAIN in to detect errors like
"Subquery returns more than 1 row", and during optimization to
perform constant optimization.
The cause for the failed ASSERT is in JOIN::join_free, where we set
bool full= (!select_lex->uncacheable && !thd->lex->describe);
Thus for EXPLAIN statements full == FALSE, and as a result the call to
JOIN::cleanup doesn't call JOIN_TAB::cleanup which should have
called table->disable_keyread().
Solution:
Consider all kinds of subquery predicates as expensive.
Merge 5.3-mwl89 into 5.3 main.
There is one remaining test failure in this merge:
innodb_mysql_lock2. All other tests have been checked to
deliver the same results/explains as 5.3-mwl89, including
the few remaining wrong results.
Phase 3: Implementation of re-optimization of subqueries with injected predicates
and cost comparison between Materialization and IN->EXISTS strategies.
The commit contains the following known problems:
- The implementation of EXPLAIN has not been re-engineered to reflect the
changes in subquery optimization. EXPLAIN for subqueries is called during
the execute phase, which results in different code paths during JOIN::optimize
and thus in differing EXPLAIN messages for constant/system tables.
- There are some valgrind warnings that need investigation
- Several EXPLAINs with minor differences need to be reconsidered after fixing
the EXPLAIN problem above.
This patch also adds one extra optimizer_switch: 'in_to_exists' for complete
manual control of the subquery execution strategies.
Fix some bugs where we stored values other than 0 or 1 in my_bool
Fixed some compiler warnings
client/mysql.cc:
Changed interrupted_query from my_bool to int, as we stored 2 in it.
client/mysqladmin.cc:
Changed return variable type to same type as function value type
client/mysqltest.cc:
Changed 'found' to int as we store other values than 0 or 1 into it
Changed type for parameter of set_reconnect() to match usage.
extra/libevent/evbuffer.c:
Added __attribute__((unused))
extra/libevent/event.c:
Added __attribute__((unused))
extra/libevent/signal.c:
Added __attribute__((unused))
sql/event_data_objects.h:
my_bool -> bool
sql/event_db_repository.cc:
my_bool -> bool
sql/event_db_repository.h:
my_bool -> bool
sql/event_parse_data.h:
my_bool -> bool
sql/events.cc:
my_bool -> bool
sql/events.h:
my_bool -> bool
sql/field.cc:
my_bool -> bool
sql/field.h:
my_bool -> bool
sql/hash_filo.h:
my_bool -> bool
sql/item.cc:
my_bool -> bool
sql/item.h:
my_bool -> bool
sql/item_cmpfunc.h:
my_bool -> bool
Changed result_for_null_param from my_bool to int as we stored -1 in it.
sql/item_func.cc:
my_bool -> bool
Modified udf wrapper functions so that the UDF functions would continue to use my_bool. (To keep compatibility with UDF:s)
sql/item_func.h:
my_bool -> bool
sql/item_subselect.h:
my_bool -> bool
sql/item_sum.cc:
Modified udf wrapper functions so that the UDF functions would continue to use my_bool. (To keep compatibility with UDF:s)
sql/parse_file.h:
my_bool -> bool
sql/rpl_mi.h:
my_bool -> bool
sql/sp_rcontext.h:
my_bool -> bool
sql/sql_analyse.h:
my_bool -> bool
sql/sql_base.cc:
Change some assignments so that we don't initialize bool variables with int's.
sql/sql_bitmap.h:
my_bool -> bool
sql/sql_cache.cc:
my_bool -> bool
sql/sql_cache.h:
my_bool -> bool
sql/sql_class.h:
my_bool -> bool
sql/sql_insert.cc:
Change some assignments so that we don't initialize bool variables with int's.
sql/sql_prepare.cc:
my_bool -> bool
sql/table.h:
my_bool -> bool
storage/maria/ma_check.c:
Removed duplicate assignment
strings/decimal.c:
Fixed wrong variable usage.
Don't do complex arithmetic on bool when simple works.
