The execution plan cannot use sorting on the first table from the
sequence of the joined tables if it plans to employ the block-based
hash join algorithm.
KEYUSE elements for a possible hash join key are not sorted by field
numbers of the second table T of the hash join operation. Besides
some of these KEYUSE elements cannot be used to build any key as their
key expressions depend on the tables that are planned to be accessed
after the table T.
The code before the patch did not take this into account and, as a result,
execition of a query the employing block-based hash join algorithm could
cause a crash or return a wrong result set.
- Make EXPLAIN display "Start temporary" at the start of the fanout (it used to display
at the first table whose rowid gets into temp. table which is not that useful for
the user)
- Updated test results (all checked)
- Break down POSITION/advance_sj_state() into four classes
representing potential semi-join strategies.
- Treat all strategies uniformly (before, DuplicateWeedout
was special as it was the catch-all strategy. Now, we're
still relying on it to be the catch-all, but are able to
function,e.g. with firstmatch=on,duplicate_weedout=off.
- Update test results (checked)
If the optimizer switch 'semijoin_with_cache' is set to 'off' then
join cache cannot be used to join inner tables of a semijoin.
Also fixed a bug in the function check_join_cache_usage() that led
to wrong output of the EXPLAIN commands for some test cases.
sql/sql_insert.cc:
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
******
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
sql/sql_table.cc:
small cleanup
******
small cleanup
- The problem was that JOIN::save/restore_query_plan() did not save/restore parts of
the query plan that are located inside SJ_MATERIALIZATION_INFO structures. This could
cause parts of one plan to be used with another, which led get_best_combination() to
constructing non-sensical join plans (and crash).
Fixed by saving/restoring SJM parts of the query plans.
- check_and_do_in_subquery_rewrites() will not set SUBS_MATERIALIZATION flag when it
records that the subquery predicate is to be converted into semi-join.
If convert_join_subqueries_to_semijoins() later decides not to convert to semi-join,
let it set SUBS_MATERIALIZATION flag, if appropriate.
- Let join buffering code correctly take into account rowids needed
by DuplicateElimination when it is calculating minimum record sizes.
- In JOIN_CACHE::write_record_data, added asserts that prevent us from
writing beyond the end of the buffer.
Analysis:
In the test query semi-join merges the inner-most subquery
into the outer subquery, and the optimization of the merged
subquery finds some new index access methods. Later the
IN-EXISTS transformation is applied to the unmerged subquery.
Since the optimizer is instructed to not consider
materialization, it reoptimizes the plan in-place to take into
account the new IN-EXISTS conditions. Just before reoptimization
JOIN::choose_subquery_plan resets the query plan, which also
resets the access methods found during the semi-join merge.
Then reoptimization discovers there are no new access methods,
but it leaves the query plan in its reset state. Later semi-join
crashes because it assumes these access methods are present.
Solution:
When reoptimizing in-place, reset the query plan only after new
access methods were discovered. If no new access methods were
discovered, leave the current plan as it was.
First code
- "Asynchronous procedure call" system
- new THD::check_killed() that serves APC request is called from within most important loops
- EXPLAIN code is now able to generate EXPLAIN output on-the-fly [incomplete]
Parts that are still missing:
- put THD::check_killed() call into every loop where we could spend significant amount of time
- Make sure EXPLAIN code works for group-by queries that replace JOIN::join_tab with make_simple_join()
and other such cases.
- User interface: what error code to use, where to get timeout settings from, etc.
The problem was that optimizer removes some outer references (it they are
constant for example) and the list of outer items built during prepare phase is
not actual during execution phase when we need it as the cache parameters.
First solution was use pointer on pointer on outer reference Item and
initialize temporary table on demand. This solved most problem except case
when optimiser also reduce Item which contains outer references ('OR' in
this bug test suite).
The solution is to build the list of outer reference items on execution
phase (after optimization) on demand (just before temporary table creation)
by walking Item tree and finding outer references among Item_ident
(Item_field/Item_ref) and Item_sum items.
Removed depends_on list (because it is not neede any mnore for the cache, in the place where it was used it replaced with upper_refs).
Added processor (collect_outer_ref_processor) and get_cache_parameters() methods to collect outer references (or other expression parameters in future).
mysql-test/r/subselect_cache.result:
A new test added.
mysql-test/r/subselect_scache.result:
Changes in creating the cache and its paremeters order or adding arguments of aggregate function (which is a parameter also, but this has no influence on the result).
mysql-test/t/subselect_cache.test:
Added a new test.
sql/item.cc:
depends_on removed.
Added processor (collect_outer_ref_processor) and get_cache_parameters() methods to collect outer references.
Item_cache_wrapper collect parameters befor initialization of its cache.
sql/item.h:
depends_on removed.
Added processor (collect_outer_ref_processor) and get_cache_parameters() methods to collect outer references.
sql/item_cmpfunc.cc:
depends_on removed.
Added processor (collect_outer_ref_processor) to collect outer references.
sql/item_cmpfunc.h:
Added processor (collect_outer_ref_processor) to collect outer references.
sql/item_subselect.cc:
depends_on removed.
Added processor get_cache_parameters() method to collect outer references.
sql/item_subselect.h:
depends_on removed.
Added processor get_cache_parameters() method to collect outer references.
sql/item_sum.cc:
Added processor (collect_outer_ref_processor) method to collect outer references.
sql/item_sum.h:
Added processor (collect_outer_ref_processor) and get_cache_parameters() methods to collect outer references.
sql/opt_range.cc:
depends_on removed.
sql/sql_base.cc:
depends_on removed.
sql/sql_class.h:
New iterator added.
sql/sql_expression_cache.cc:
Build of list of items resolved in outer query done just before creating expression cache on the first execution of the subquery which removes influence of optimizer removing items (all optimization already done).
sql/sql_expression_cache.h:
Build of list of items resolved in outer query done just before creating expression cache on the first execution of the subquery which removes influence of optimizer removing items (all optimization already done).
sql/sql_lex.cc:
depends_on removed.
sql/sql_lex.h:
depends_on removed.
sql/sql_list.h:
Added add_unique method to add only unique elements to the list.
sql/sql_select.cc:
Support of new Item list added.
sql/sql_select.h:
Support of new Item list added.
Also:
1. simplified the code of the function mysql_derived_merge_for_insert.
2. moved merge of views/dt for multi-update/delete to the prepare stage.
3. the list of the references to the candidates for semi-join now is
allocated in the statement memory.
The bitmap of used tables must be evaluated for the select list of every
materialized derived table / view and saved in a dedicated field.
This is also applied to materialized subqueries.
- Don't attempt to construct FirstMatch access method if we've
just figured three lines above that it can't be used (because join
prefix doesn't have the needed tables), and so have set
pos->first_firstmatch_table= MAX_TABLES
Attempts to analyze join->positions[MAX_TABLES] caused valgrind warnings
mysql-test/r/subselect4.result:
Moved test case for LP BUG#718593 into the correct test file subselect_mat_cost_bugs.test.
mysql-test/t/subselect4.test:
Moved test case for LP BUG#718593 into the correct test file subselect_mat_cost_bugs.test.
Resolved all conflicts, bad merges and fixed a few minor bugs in the code.
Commented out the queries from multi_update, view, subselect_sj, func_str,
derived_view, view_grant that failed either with crashes in ps-protocol or
with wrong results.
The failures are clear indications of some bugs in the code and these bugs
are to be fixed.
sql/item_subselect.cc:
Cleanup. Comments added.
sql/item_subselect.h:
Cleanup.
sql/mysql_priv.h:
Comments added.
sql/opt_subselect.cc:
The function renamed and turned to method.
Comments added.
sql/opt_subselect.h:
The function turned to method of JOIN.
sql/sql_select.cc:
Comment added. The function turned to method.
sql/sql_select.h:
The function turned to method.
mysql-test/r/explain.result:
fixed results (new item)
mysql-test/r/subselect.result:
fixed results (new item)
mysql-test/r/subselect_no_mat.result:
fixed results (new item)
mysql-test/r/subselect_no_opts.result:
fixed results (new item)
mysql-test/r/subselect_no_semijoin.result:
Fixed results (new item)
mysql-test/suite/pbxt/r/subselect.result:
Fixed results (new item)
mysql-test/t/explain.test:
Fixed results (correct behaviour)
sql/item_cmpfunc.cc:
Pass through for max/min
sql/item_subselect.cc:
moving max/min
sql/item_subselect.h:
moving max/min
sql/mysql_priv.h:
new uncacheble flags added
sql/opt_subselect.cc:
maxmin moved.
sql/opt_subselect.h:
New function for maxmin.
sql/sql_class.h:
debug code
sql/sql_lex.cc:
Fixed flags.
Limit setting fixed.
sql/sql_lex.h:
2 new flags.
sql/sql_select.cc:
Prepare divided on 2 function to be able recollect some info after transformation.
sql/sql_select.h:
Prepare divided on 2 functions.
- Let advance_sj_state() save the value of JOIN::cur_dups_producing_tables
in POSITION::prefix_dups_producing_tables, and restore_sj_state() restore
it.
Valgrind warnings were caused by comparing index values to an un-initialized field.
mysql-test/r/subselect.result:
New test cases.
mysql-test/t/subselect.test:
New test cases.
sql/opt_sum.cc:
Add thd to opt_sum_query enabling it to test for errors.
If we have a non-nullable index, we cannot use it to match null values,
since set_null() will be ignored, and we might compare uninitialized data.
sql/sql_select.cc:
Add thd to opt_sum_query, enabling it to test for errors.
sql/sql_select.h:
Add thd to opt_sum_query, enabling it to test for errors.
Analysis:
There are two code paths through which JOIN::exec may produce
an all-NULL row for an empty result set. One goes via the
function return_zero_rows(), when query processing detectes
early that the where clause is false, the other one is via
do_select() in the case of join execution.
In the case of do_select(), the problem was that the executioner
didn't set TABLE::null_row to 1. As result when sending the only
result row, the evaluation of each field didn't detect that all
non-aggregated fields are NULL, because Field::is_null returned
true, after checking that field->table->null_row was false.
Given that the each non-aggregated field was not considered NULL,
select_result::send_data sent whatever was in the buffer of each
field. However, since there was no actual data in the field buffer,
send_data() accessed and sent whatever junk was in the field's
data buffer.
Solution:
Similar to the analogous case in return_zero_rows() mark all
tables that their current row is NULL before sending the
artificailly created NULL row.
If join condition is of the form <t2.key>=<t1.no_key> then the server
performs no index look-ups when looking for matching rows of t2 for
the rows from t1 with t1.no_key=NULL. It happens because the function
add_not_null_conds() injects an additional condition of the form
IS NOT NULL(<t1.no_key>) into the WHERE condition.
However if the join condition was of the form <t.key>=<outer_ref> no
additional null rejecting predicate was generated. This could lead
to extra records in the result set if the value of <outer_ref> happened
to be NULL.
The new code injects null rejecting predicates of the form
IS NOT NULL(<outer_ref>) and evaluates them before the first row
the subquery is constructed.
- put the code that sets HA_NULL_PART bit back
- Fix test_if_ref/part_of_refkey() so that
= NULL-ability of lookup columns does not prevent the equality
from being removed (we now have early/late NULLs filtering which
will filter out NULL values)
= equality is not removed if it is ref_or_null access, and the value
of the lookup column can alternate between the lookup value and NULL.
even in the cases when there existed range/index-merge scans that
were cheaper than the full table scan.
This was a defect/bug of the implementation of mwl #128.
Now hash join can work not only with full table scan of the joined
table, but also with full index scan, range and index-merge scans.
Accordingly, in the cases when hash join is used the column 'type'
in the EXPLAINs can contain now 'hash_ALL', 'hash_index', 'hash_range'
and 'hash_index_merge'. If hash join is coupled with a range/index_merge
scan then the columns 'key' and 'key_len' contain info not only on
the used hash index, but also on the indexes used for the scan.
- Fixed some issues with partitions and connection_string, which also fixed lp:716890 "Pre- and post-recovery crash in Aria"
- Fixed wrong assert in Aria
Now need to merge with latest xtradb before pushing
sql/ha_partition.cc:
Ensure that m_ordered_rec_buffer is not freed before close.
sql/mysqld.cc:
Changed to use opt_stack_trace instead of opt_pstack.
Removed references to pstack
sql/partition_element.h:
Ensure that connect_string is initialized
storage/maria/ma_key_recover.c:
Fixed wrong assert
The bug was in the code of the patch fixing bug 698882.
With improper casting the method store_key_field::change_source_field
was called for the elements of the array TABLE_REF::key_copy that
were either of a different type or not allocated at all. This caused
crashes in some queries.
Made sure that the optimal fields are used by TABLE_REF objects
when building index access keys to joined tables.
Fixed a bug in the template function that sorts the elements of
a list using the bubble sort algorithm. The bug caused poor
performance of the function. Also added an optimization that
skips comparison with the most heavy elements that has been
already properly placed in the list.
Made the comparison of the fields belonging to the same Item_equal
more granular: fields belonging to the same table are also ordered
according to some rules.
- Removed files specific to compiling on OS/2
- Removed files specific to SCO Unix packaging
- Removed "libmysqld/copyright", text is included in documentation
- Removed LaTeX headers for NDB Doxygen documentation
- Removed obsolete NDB files
- Removed "mkisofs" binaries
- Removed the "cvs2cl.pl" script
- Changed a few GPL texts to use "program" instead of "library"
Analysis:
The fix for LP BUG#680846 avoids evaluation of constant expressions
with subqueries in the GROUP/ORDER clauses in the procedure
remove_const(). The purpose of remove_const is to remove constant
expressions in the GROUP/ORDER clauses.
In order delay until execution the evaluation of such subqueries,
they were not removed in the GROUP/ORDER clause. As a result temp
table creation during execution attempted to create a column in the
temp table for each constant GROUP/ORDER expression. However, the
logic in create_tmp_table is to not create temp table columns for
constant items. The crash was due to a group Item without a
corresponding column in the temp table for GROUP BY.
Solution:
The patch adds back removal of constant expressions with subqueries.
In order for such expressions to be evaluated, so that the server can
ensure that such subquries return 1 row, the evaluation of these
expressions is delayed until execution.
This is a backport of the fix for
MySQL BUG#52317: Assertion failing in Field_varstring::store () at field.cc:6833
The orginal comment by Oystein is:
In order for EXPLAIN to print const-refs, a Store_key_const_item object
is created. This is different for normal execution of subqueries where
a temporary store_key_item object is used instead. The problem is that
EXPLAIN will execute subqueries. This leads to a scenario where a
store_key_const_item object it told to write to its underlying field.
This results in a failing assert since the write set of the underlying
table does not reflect this.
The resolution is to do the same trick as for store_key_item::copy_inner().
That is, temporarily change the write set to allow writes to all columns.
This is only necessary in debug version since non-debug version does not
contain asserts on write_set.
sql/sql_select.h:
Temporarily change write_set in store_key_const_item::copy_inner() to
allow initialization of underlying field. This is necessary since
subqueries are executed for EXPLAIN. (For normal execution,
store_key_item::copy_inner is used.)
The condition that was supposed to check whether a join table
is an inner table of a nested outer join or semi-join was not
quite correct in the code of the function check_join_cache_usage.
That's why some queries with nested outer joins triggered
an assertion failure.
Encapsulated this condition in the new method called
JOIN_TAB::is_nested_inner and provided a proper code for it.
Also corrected a bug in the code of check_join_cache_usage()
that caused a downgrade of not first join buffers of the
level 5 and 7 to level 4 and 6 correspondingly.
The cause for the bug was two-fold:
1. Incorrect detection of whether a table is the first one in a query plan -
"used_table & 1" actually checks if used_table is table with number "1".
2. Missing logic to delay the evaluation of (expensive) constant conditions
during the execution phase.
The fix adds/changes:
The patch:
- removes incorrect treatment of expensive predicates from make_cond_for_table,
and lets the caller decide when to evaluate expensive predicates.
- saves expensive constant conditions in JOIN::exec_const_cond,
which is evaluated once in the beginning of JOIN::exec.
A non-incremental join buffer cannot be used for inner tables of nested
outer joins. That's why when join_cache_level is set to 7 it must
be downgraded to level 6 for the inner tables of nested outer joins.
For the same reason with join_cache_level set to 3 no join buffer is
used for the inner tables of outer joins (we could downgrade it to
level 2, but this level does not support ref access).
Currently BNLH join uses a simplified implementation of hash function
when hash function is calculated over the whole key buffer, not only
the significant bytes of it. It means that both building keys and
probing keys both must fill insignificant bytes with the same filler.
Usually 0 is used as such a filler.
Yet the code before patch filled insignificant bytes only for probing
keys.
