CHEAP SQ: Valgrind warnings "Memory lost" with IN and EXISTS nested subquery, materialization+semijoin
Analysis:
The memory leak was a result of the interaction of semi-join optimization
with early optimization of constant subqueries. The function:
setup_jtbm_semi_joins() created a dummy temporary table "dummy_table"
in order to make some JOIN_TAB objects complete. Normally, such temporary
tables are freed inside JOIN_TAB::cleanup.
However, the inner-most subquery is pre-optimized, which allows the
optimization fo the MAX subquery to determine that its WHERE is TRUE,
and thus to compute the result of the MAX during optimization. This
ultimately allows the optimize phase of the outer query to find that
it WHERE clause is FALSE. Once JOIN::optimize finds that the result
set is empty, it sets zero_result_cause, and returns *before* it ever
reached make_join_statistics(). As a result the query plan has no
JOIN_TABs at all. Since the temporary table is supposed to be cleanup
via JOIN_TAB::cleanup, this never happens because there is no JOIN_TAB
for this table. Hence we get a memory leak.
Solution:
Whenever there are no JOIN_TABs, iterate over all table reference in
JOIN::join_list, and free the ones that contain semi-join temporary
tables.
Analysis:
The fix for lp:944706 introduces early subquery optimization.
While a subquery is being optimized some of its predicates may be
removed. In the test case, the EXISTS subquery is constant, and is
evaluated to TRUE. As a result the whole OR is TRUE, and thus the
correlated condition "b = alias1.b" is optimized away. The subquery
becomes non-correlated.
The subquery cache is designed to work only for correlated subqueries.
If constant subquery optimization is disallowed, then the constant
subquery is not evaluated, the subquery remains correlated, and its
execution is cached. As a result execution is fast.
However, when the constant subquery was optimized away, it was neither
cached by the subquery cache, nor it was cached by the internal subquery
caching. The latter was due to the fact that the subquery still appeared
as correlated to the subselect_XYZ_engine::exec methods, and they
re-executed the subquery on each call to Item_subselect::exec.
Solution:
The solution is to update the correlated status of the subquery after it has
been optimized. This status consists of:
- st_select_lex::is_correlated
- Item_subselect::is_correlated
- SELECT_LEX::uncacheable
- SELECT_LEX_UNIT::uncacheable
The status is updated by st_select_lex::update_correlated_cache(), and its
caller st_select_lex::optimize_unflattened_subqueries. The solution relies
on the fact that the optimizer already called
st_select_lex::update_used_tables() for each subquery. This allows to
efficiently update the correlated status of each subquery without walking
the whole subquery tree.
Notice that his patch is an improvement over MySQL 5.6 and older, where
subqueries are not pre-optimized, and the above analysis is not possible.
Analysis:
The problem in the original MySQL bug is that the range optimizer
performs its analysis in a separate MEM_ROOT object that is freed
after the range optimzier is done. During range analysis get_mm_tree
calls Item_func_like::select_optimize, which in turn evaluates its
right argument. In the test case the right argument is a subquery.
In MySQL, subqueries are optimized lazyly, thus the call to val_str
triggers optimization for the subquery. All objects needed by the
subquery plan end up in the temporary MEM_ROOT used by the range
optimizer. When execution ends, the JOIN::cleanup process tries to
cleanup objects of the subquery plan, but all these objects are gone
with the temporary MEM_ROOT. The solution for MySQL is to switch the
mem_root.
In MariaDB with the patch for bug lp:944706, all constant subqueries
that may be used by the optimization process are preoptimized. Therefore
Item_func_like::select_optimize only triggers subquery execution, and
the above problem is not present.
The patch however adds a test whether the evaluated right argument of
the LIKE predicate is expensive. This is consistent with our approach
not to evaluate expensive expressions during optimization.
The cause for this bug is that the method JOIN::get_examined_rows iterates over all
JOIN_TABs of the join assuming they are just a sequence. In the query above, the
innermost subquery is merged into its parent query. When we call
JOIN::get_examined_rows for the second-level subquery, the iteration that
assumes sequential order of join tabs goes outside the join_tab array and calls
the method JOIN_TAB::get_examined_rows on uninitialized memory.
The fix is to iterate over JOIN_TABs in a way that takes into account the nested
semi-join structure of JOIN_TABs. In particular iterate as select_describe.
The patch enables back constant subquery execution during
query optimization after it was disabled during the development
of MWL#89 (cost-based choice of IN-TO-EXISTS vs MATERIALIZATION).
The main idea is that constant subqueries are allowed to be executed
during optimization if their execution is not expensive.
The approach is as follows:
- Constant subqueries are recursively optimized in the beginning of
JOIN::optimize of the outer query. This is done by the new method
JOIN::optimize_constant_subqueries(). This is done so that the cost
of executing these queries can be estimated.
- Optimization of the outer query proceeds normally. During this phase
the optimizer may request execution of non-expensive constant subqueries.
Each place where the optimizer may potentially execute an expensive
expression is guarded with the predicate Item::is_expensive().
- The implementation of Item_subselect::is_expensive has been extended
to use the number of examined rows (estimated by the optimizer) as a
way to determine whether the subquery is expensive or not.
- The new system variable "expensive_subquery_limit" controls how many
examined rows are considered to be not expensive. The default is 100.
In addition, multiple changes were needed to make this solution work
in the light of the changes made by MWL#89. These changes were needed
to fix various crashes and wrong results, and legacy bugs discovered
during development.
The problem was increment of aborted_threads variable due to thd->killed which was set when threadpool connection was terminated . The fix is not to set thd->killed anymore, there is no real reason for doing it..
Added a test that checks that status variable aborted_clients does not grow for ordinary disconnects, and that successful KILL increments this variable.
The problem was in the code (update_const_equal_items()) which marked
index parts constant independently of the place where the equality was used.
In the test suite it marked t2_1.c part constant despite the fact that
it connected by OR with other expression.
Solution is to mark constant only top equalities connected with AND.
Create an Item_cache based on item's cmp_type, not result_type in
subselect_engine.
Use result_field in Item_cache_temporal::cache_value(),
just like all other Item_cache*::cache_value() do.
Points and lines should disappear if we got negative D.
To make it work properly inside the GEOMETRYCOLLECTION,
we add the empty operation there.
bug #986977 Assertion `!cur_p->event' failed in Gcalc_scan_iterator::arrange_event(int, int).
The double->inernal coord conversion produced -0 (minus zero) on some data.
That minus-zero produces invalid comparison results when compared agains plus-zero.
So we fixed the gcalc_set_double() to avoid it.
per-file comments:
mysql-test/r/gis-precise.result
result updated.
mysql-test/t/gis-precise.test
tests for #977021 and #986977 added.
sql/gcalc_slicescan.cc
bug #986977. The gcalc_set_double fixed to not produce minus-zero.
sql/item_geofunc.cc
bug #977021. Add the NOOP for the disappearing features.
Analysis:
The reason for the wrong result is the interaction between constant
optimization (in this case 1-row table) and subquery optimization.
- First the outer query is optimized, and 'make_join_statistics' finds that
table t2 has one row, reads that row, and marks the whole table as constant.
This also means that all fields of t2 are constant.
- Next, we optimize the subquery in the end of the outer 'make_join_statistics'.
The field 'f2' is considered constant, with value '3'. The subquery predicate
is rewritten as the constant TRUE.
- The outer query execution detects early that the whole query result is empty
and calls 'return_zero_rows'. Since the query is with implicit grouping, we
have to produce one row with special values for the aggregates (depending on
each aggregate function), and NULL values for all non-aggregate fields. This
function calls 'no_rows_in_result' to set each aggregate function to the
default value when it aggregates over an empty result, and then calls
'send_data', which in turn evaluates each Item in the SELECT list.
- When evaluation reaches the subquery predicate, it executes the subquery
with field 'f2' having a constant value '3', and the subquery produces the
incorrect result '7'.
Solution:
Implement Item::no_rows_in_result for all subquery predicates. In order to
make this work, it is also needed to make all val_* methods of all subquery
predicates respect the Item_subselect::forced_const flag. Otherwise subqueries
are executed anyways, and override the default value set by no_rows_in_result
with whatever result is produced from the subquery evaluation.
Points and lines should disappear if we got negative D.
To make it work properly inside the GEOMETRYCOLLECTION,
we add the empty operation there.
bug #986977 Assertion `!cur_p->event' failed in Gcalc_scan_iterator::arrange_event(int, int).
The double->inernal coord conversion produced -0 (minus zero) on some data.
That minus-zero produces invalid comparison results when compared agains plus-zero.
So we fixed the gcalc_set_double() to avoid it.
per-file comments:
mysql-test/r/gis-precise.result
result updated.
mysql-test/t/gis-precise.test
tests for #977021 and #986977 added.
sql/gcalc_slicescan.cc
bug #986977. The gcalc_set_double fixed to not produce minus-zero.
sql/item_geofunc.cc
bug #977021. Add the NOOP for the disappearing features.
Fixed incorrect type casting which made all fields (except very first) changes to materialized table incorrect.
Saved list of view/derived table used items after expanding '*'.
Part#1: make EXPLAIN's plan match the one by actual execution:
Item_subselect::used_tables() should return the same value irrespectively
of whether we're running an EXPLAIN or a SELECT.
When the function free_tmp_table deletes the handler object for
a temporary table the field TABLE::file for this table should be
set to NULL. Otherwise an assertion failure may occur.
This bug happened because the function find_field_in_view formed
autogenerated names of view columns without a possibility to roll
them back. In some situation it could cause memory misuses reported
by valgrind or even crashes.
When a view/derived table is converted from merged to materialized the
items from the used_item lists are substituted for items referring to
the fields of the result of the materialization. The problem appeared
with queries employing natural joins. Since the resolution of a natural
join was performed only once the used_item list formed at the second
execution of the query lacked the references to the fields that were
used only in the equality predicates generated for the natural join.
Moved test from main suite to the new suites.
Move tests from maria/t and maria/r to maria
mysql-test/mysql-test-run.pl:
Added support for the new suites
The main problem was a bug in CSV where it provided wrong statistics (it claimed the table was empty when it wasn't)
I also fixed wrong freeing of blob's in the CSV handler. (Any call to handler::read_first_row() on a CSV table with blobs would fail)
mysql-test/r/csv.result:
Added new test case
mysql-test/r/partition_innodb.result:
Updated test results after fixing bug with impossible partitions and const tables
mysql-test/t/csv.test:
Added new test case
sql/sql_select.cc:
Cleaned up code for handling of partitions.
Fixed also a bug where we didn't threat a table with impossible partitions as a const table.
storage/csv/ha_tina.cc:
Allocate blobroot onces.
Background:
- as described in MySQL Internals Prepared Stored
(http://forge.mysql.com/wiki/MySQL_Internals_Prepared_Stored),
the Optimizer sometimes does destructive changes to the parsed
LEX-object (Item-tree), which makes it impossible to re-use
that tree for PS/SP re-execution.
- in order to be able to re-use the Item-tree, the destructive
changes are remembered and rolled back after the statement execution.
The problem, discovered by this bug, was that the objects representing
GROUP-BY clause did not restored after query execution. So, the GROUP-BY
part of the statement could not be properly re-initialized for re-execution
after destructive changes.
Those objects do not take part in the Item-tree, so they can not be saved
using the approach for Item-tree.
The fix is as follows:
- introduce a new array in st_select_lex to store the original
ORDER pointers, representing the GROUP-BY clause;
- Initialize this array in fix_prepare_information().
- restore the list of GROUP-BY items in reinit_stmt_before_use().
Analysis:
-------------------------------
According to the Manual
(http://dev.mysql.com/doc/refman/5.1/en/identifier-case-sensitivity.html):
"Column, index, stored routine, and event names are not case sensitive on any
platform, nor are column aliases."
In other words, 'lower_case_table_names' does not affect the behaviour of
those identifiers.
On the other hand, trigger names are case sensitive on some platforms,
and case insensitive on others. 'lower_case_table_names' does not affect
the behaviour of trigger names either.
The bug was that SHOW statements did case sensitive comparison
for stored procedure / stored function / event names.
Fix:
Modified the code so that comparison in case insensitive for routines
and events for "SHOW" operation.
As part of this commit, only fixing the test failures due to the actual code fix.
mysql-test/suite/innodb/t/group_commit_crash.test:
remove autoincrement to avoid rbr being used for insert ... select
mysql-test/suite/innodb/t/group_commit_crash_no_optimize_thread.test:
remove autoincrement to avoid rbr being used for insert ... select
mysys/my_addr_resolve.c:
a pointer to a buffer is returned to the caller -> the buffer cannot be on the stack
mysys/stacktrace.c:
my_vsnprintf() is ok here, in 5.5
