This is a backport of the fix for MySQL bug #13723054 in 5.6.
Original comment:
The crash is caused by arbitrary memory area owerwriting in case of
BLOB fields during attempt to copy BLOB field key image into record
buffer(record buffer is too small to get BLOB key part image).
note:
QUICK_GROUP_MIN_MAX_SELECT can not work with BLOB fields
because it uses record buffer as temporary buffer for key values
however this case is filtered out by covering_keys() check
in get_best_group_min_max() as BLOBs always require key length
modificator in the key declaration and if the key has a BLOB
then it can not be covered key.
The fix is to use 'max_used_key_length' key length instead of 0.
Analysis:
Spcifically the crash in this bug was a result of the call to key_copy()
that copied the whole key, inlcuding the BLOB field which is not used
for index access. Copying the blob field overwrote memory as far as the
function parameter 'key_info'. As a result the contents of key_info was
all 0, which resulted in a crash when this key_info was accessed few
lines below in key_cmp().
This bug led to wrong values of the use_count fields in some SEL_ARG
trees that triggered complains on the server side when executing the
test case for LP bug 800184 if a debug build of the server was used.
This was the result of the incomplete fix for bug 800184.
To complete it the following corrections had to be made:
- the copy constructor for SEL_TREE must call the new function incr_refs_all()
instead of the function incr_refs(), because references to next key parts
from any SEL_ARG tree belonging to the list of the first key part has to be
adjusted.
- the method and_sel_tree of the class SEL_IMERGE must use the copy constructor
of the SEL_TREE class to make a copy of its second argument before it ANDs it
with any SEL_TREE tree from the processed SEL_IMERGE object.
The range optimizer incorrectly chose a loose scan for group by
when there is a correlated WHERE condition. This range access
method cannot be executed for correlated conditions also with the
"range checked for each record" because generally the range access
method can change for each outer record. Loose scan destructively
changes the query plan and removes the GROUP operation, which will
result in wrong query plans if another range access is chosen
dynamically.
Analysis:
The class member QUICK_GROUP_MIN_MAX_SELECT::seen_first_key
was not reset between subquery re-executions. Thus each
subsequent execution continued from the group that was
reached by the previous subquery execution. As a result
loose scan reached end of file much earlier, and returned
empty result where it shouldn't.
Solution:
Reset seen_first_key before each re-execution of the
loose scan.
Due to this bug the function SEL_IMERGE::or_sel_tree_with_checks()
could build an inconsistent merge tree if one of the SEL_TREEs in the
resulting index merge happened to contain a full key range.
This could trigger an assertion failure.
The function key_and() erroneously called SEL_ARG::increment_use_count()
when SEL_ARG::incr_refs() should had been called. This could lead to
wrong values of use_count for some SEL_ARG trees.
A bug in the code of the function key_or could lead to a situation
when performing of an OR operation for one index changes the result
the operation for another index. This bug is fixed with this patch.
Also corrected the specification and the code of the function
or_sel_tree_with_checks.
sql/sql_insert.cc:
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
******
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
sql/sql_table.cc:
small cleanup
******
small cleanup
The assertion in innodb is triggered in this way:
1. mysql server does lookup on the primary key with full key,
innodb decides to not store cursor position because
"any index_next/prev call will return EOF anyway"
2. server asks innodb to return any next record in the index and the
assertion is triggered because no cursor position is stored.
It happens when a unique search (match_mode=ROW_SEL_EXACT)
in the clustered index is performed. InnoDB has never stored
the cursor position after a unique key lookup in the
clustered index because storing the position is an expensive
operation. The bug was introduced by
WL3220 'Loose index scan for aggregate functions'.
The fix is to disallow loose index scan optimization
for AGG_FUNC(DISTINCT ...) if GROUP_MIN_MAX quick select
uses clustered key.
mysql-test/r/group_min_max_innodb.result:
test case
mysql-test/t/group_min_max_innodb.test:
test case
sql/opt_range.cc:
disallow loose index scan optimization for
AGG_FUNC(DISTINCT ...) if GROUP_MIN_MAX
quick select uses clustered key.
opt_range.cc: modified print_key() so that it doesn't do memory re-allocs when
printing multipart keys over varchar columns. When it did, key printout in
debug trace was interrupted with my_malloc/free printouts.
revno: 2876.47.174
revision-id: jorgen.loland@oracle.com-20110519120355-qn7eprkad9jqwu5j
parent: mayank.prasad@oracle.com-20110518143645-bdxv4udzrmqsjmhq
committer: Jorgen Loland <jorgen.loland@oracle.com>
branch nick: mysql-trunk-11765831
timestamp: Thu 2011-05-19 14:03:55 +0200
message:
BUG#11765831: 'RANGE ACCESS' MAY INCORRECTLY FILTER
AWAY QUALIFYING ROWS
The problem was that the ranges created when OR'ing two
conditions could be incorrect. Without the bugfix,
"I <> 6 OR (I <> 8 AND J = 5)" would create these ranges:
"NULL < I < 6",
"6 <= I <= 6 AND 5 <= J <= 5",
"6 < I < 8",
"8 <= I <= 8 AND 5 <= J <= 5",
"8 < I"
While the correct ranges is
"NULL < I < 6",
"6 <= I <= 6 AND 5 <= J <= 5",
"6 < I"
The problem occurs when key_or() ORs
(1) "NULL < I < 6, 6 <= I <= 6 AND 5 <= J <= 5, 6 < I" with
(2) "8 < I AND 5 <= J <= 5"
The reason for the bug is that in key_or(), SEL_ARG *tmp is
used to point to the range in (1) above that is merged with
(2) while key1 points to the root of the red-black tree of
(1). When merging (1) and (2), tmp refers to the "6 < I"
part whereas the root is the "6 <= ... AND 5 <= J <= 5" part.
key_or() decides that the tmp range needs to be split into
"6 < I < 8, 8 <= I <= 8, 8 < I", in which next_key_part of the
second range should be that of tmp. However, next_key_part is
set to key1->next_key_part ("5 <= J <= 5") instead of
tmp->next_key_part (empty). Fixing this gives the correct but
not optimal ranges:
"NULL < I < 6",
"6 <= I <= 6 AND 5 <= J <= 5",
"6 < I < 8",
"8 <= I <= 8",
"8 < I"
A second problem can be seen above: key_or() may create
adjacent ranges that could be replaced with a single range.
Fixes for this is also included in the patch so that the range
above becomes correct AND optimal:
"NULL < I < 6",
"6 <= I <= 6 AND 5 <= J <= 5",
"6 < I"
Merging adjacent ranges like this gives a slightly lower cost
estimate for the range access.
The problem was that optimizer removes some outer references (it they are
constant for example) and the list of outer items built during prepare phase is
not actual during execution phase when we need it as the cache parameters.
First solution was use pointer on pointer on outer reference Item and
initialize temporary table on demand. This solved most problem except case
when optimiser also reduce Item which contains outer references ('OR' in
this bug test suite).
The solution is to build the list of outer reference items on execution
phase (after optimization) on demand (just before temporary table creation)
by walking Item tree and finding outer references among Item_ident
(Item_field/Item_ref) and Item_sum items.
Removed depends_on list (because it is not neede any mnore for the cache, in the place where it was used it replaced with upper_refs).
Added processor (collect_outer_ref_processor) and get_cache_parameters() methods to collect outer references (or other expression parameters in future).
mysql-test/r/subselect_cache.result:
A new test added.
mysql-test/r/subselect_scache.result:
Changes in creating the cache and its paremeters order or adding arguments of aggregate function (which is a parameter also, but this has no influence on the result).
mysql-test/t/subselect_cache.test:
Added a new test.
sql/item.cc:
depends_on removed.
Added processor (collect_outer_ref_processor) and get_cache_parameters() methods to collect outer references.
Item_cache_wrapper collect parameters befor initialization of its cache.
sql/item.h:
depends_on removed.
Added processor (collect_outer_ref_processor) and get_cache_parameters() methods to collect outer references.
sql/item_cmpfunc.cc:
depends_on removed.
Added processor (collect_outer_ref_processor) to collect outer references.
sql/item_cmpfunc.h:
Added processor (collect_outer_ref_processor) to collect outer references.
sql/item_subselect.cc:
depends_on removed.
Added processor get_cache_parameters() method to collect outer references.
sql/item_subselect.h:
depends_on removed.
Added processor get_cache_parameters() method to collect outer references.
sql/item_sum.cc:
Added processor (collect_outer_ref_processor) method to collect outer references.
sql/item_sum.h:
Added processor (collect_outer_ref_processor) and get_cache_parameters() methods to collect outer references.
sql/opt_range.cc:
depends_on removed.
sql/sql_base.cc:
depends_on removed.
sql/sql_class.h:
New iterator added.
sql/sql_expression_cache.cc:
Build of list of items resolved in outer query done just before creating expression cache on the first execution of the subquery which removes influence of optimizer removing items (all optimization already done).
sql/sql_expression_cache.h:
Build of list of items resolved in outer query done just before creating expression cache on the first execution of the subquery which removes influence of optimizer removing items (all optimization already done).
sql/sql_lex.cc:
depends_on removed.
sql/sql_lex.h:
depends_on removed.
sql/sql_list.h:
Added add_unique method to add only unique elements to the list.
sql/sql_select.cc:
Support of new Item list added.
sql/sql_select.h:
Support of new Item list added.
Analysis:
This bug consists of two related problems that are
result of too early evaluation of single-row subqueries
during the optimization phase of the outer query.
Several optimizer code paths try to evaluate single-row
subqueries in order to produce a constant and use that
constant for further optimzation.
When the execution of the subquery peforms destructive
changes to the representation of the subquery, and these
changes are not anticipated by the subsequent optimization
phases of the outer query, we tipically get a crash or
failed assert.
Specifically, in this bug the inner-most suqbuery with
DISTINCT triggers a substitution of the original JOIN
object by a single-table JOIN object with a temp table
needed to perform the DISTINCT operation (created by
JOIN::make_simple_join).
This substitution breaks EXPLAIN because:
a) in the first example JOIN::cleanup no longer can
reach the original table of the innermost subquery, and
close all indexes, and
b) in this second test query, EXPLAIN attempts to print
the name of the internal temp table, and crashes because
the temp table has no name (NULL pointer instead).
Solution:
a) fully disable subquery evaluation during optimization
in all cases - both for constant propagation and range
optimization, and
b) change JOIN::join_free() to perform cleanup irrespective
of EXPLAIN or not.
Split status variable Rows_read to Rows_read and Rows_tmp_read so that one can see how much real data is read.
Same was done with with Handler_update and Handler_write.
Fixed bug in MEMORY tables where some variables was counted twice.
Added new internal handler call 'ha_close()' to have one place to gather statistics.
Fixed bug where thd->open_options was set to wrong value when doing admin_recreate_table()
mysql-test/r/status.result:
Updated test results and added new tests
mysql-test/r/status_user.result:
Udated test results
mysql-test/t/status.test:
Added new test for temporary table status variables
sql/ha_partition.cc:
Changed to call ha_close() instead of close()
sql/handler.cc:
Added internal_tmp_table variable for easy checking of temporary tables.
Added new internal handler call 'ha_close()' to have one place to gather statistics.
Gather statistics for internal temporary tables.
sql/handler.h:
Added handler variables internal_tmp_table, rows_tmp_read.
Split function update_index_statistics() to two.
Added ha_update_tmp_row() for faster tmp table handling with more statistics.
sql/item_sum.cc:
ha_write_row() -> ha_write_tmp_row()
sql/multi_range_read.cc:
close() -> ha_close()
sql/mysqld.cc:
New status variables: Rows_tmp_read, Handler_tmp_update and Handler_tmp_write
sql/opt_range.cc:
close() -> ha_close()
sql/sql_base.cc:
close() -> ha_close()
sql/sql_class.cc:
Added handling of rows_tmp_read
sql/sql_class.h:
Added new satistics variables.
rows_read++ -> update_rows_read() to be able to correctly count reads to internal temp tables.
Added handler::ha_update_tmp_row()
sql/sql_connect.cc:
Added comment
sql/sql_expression_cache.cc:
ha_write_row() -> ha_write_tmp_row()
sql/sql_select.cc:
close() -> ha_close()
ha_update_row() -> ha_update_tmp_row()
sql/sql_show.cc:
ha_write_row() -> ha_write_tmp_row()
sql/sql_table.cc:
Fixed bug where thd->open_options was set to wrong value when doing admin_recreate_table()
sql/sql_union.cc:
ha_write_row() -> ha_write_tmp_row()
sql/sql_update.cc:
ha_write_row() -> ha_write_tmp_row()
sql/table.cc:
close() -> ha_close()
storage/heap/ha_heap.cc:
Removed double counting of statistic variables.
close -> ha_close() to get tmp table statistics.
storage/maria/ha_maria.cc:
close -> ha_close() to get tmp table statistics.
introduced at the latest merge 5.1->5.2->5.3.
It is basically not needed since if SQL_SELECT::pre_idx_push_select_cond
is not NULL then SQL_SELECT::original_cond would point to the same condition
as SQL_SELECT::pre_idx_push_select_cond. Otherwise SQL_SELECT::original_cond
would be equal to SQL_SELECT::cond.
A lot of small fixes and new test cases.
client/mysqlbinlog.cc:
Cast removed
client/mysqltest.cc:
Added missing DBUG_RETURN
include/my_pthread.h:
set_timespec_time_nsec() now only takes one argument
mysql-test/t/date_formats.test:
Remove --disable_ps_protocl as now also ps supports microseconds
mysys/my_uuid.c:
Changed to use my_interval_timer() instead of my_getsystime()
mysys/waiting_threads.c:
Changed to use my_hrtime()
sql/field.h:
Added bool special_const_compare() for fields that may convert values before compare (like year)
sql/field_conv.cc:
Added test to get optimal copying of identical temporal values.
sql/item.cc:
Return that item_int is equal if it's positive, even if unsigned flag is different.
Fixed Item_cache_str::save_in_field() to have identical null check as other similar functions
Added proper NULL check to Item_cache_int::save_in_field()
sql/item_cmpfunc.cc:
Don't call convert_constant_item() if there is nothing that is worth converting.
Simplified test when years should be converted
sql/item_sum.cc:
Mark cache values in Item_sum_hybrid as not constants to ensure they are not replaced by other cache values in compare_datetime()
sql/item_timefunc.cc:
Changed sec_to_time() to take a my_decimal argument to ensure we don't loose any sub seconds.
Added Item_temporal_func::get_time() (This simplifies some things)
sql/mysql_priv.h:
Added Lazy_string_decimal()
sql/mysqld.cc:
Added my_decimal constants max_seconds_for_time_type, time_second_part_factor
sql/table.cc:
Changed expr_arena to be of type CONVENTIONAL_EXECUTION to ensure that we don't loose any items that are created by fix_fields()
sql/tztime.cc:
TIME_to_gmt_sec() now sets *in_dst_time_gap in case of errors
This is needed to be able to detect if timestamp is 0
storage/maria/lockman.c:
Changed from my_getsystime() to set_timespec_time_nsec()
storage/maria/ma_loghandler.c:
Changed from my_getsystime() to my_hrtime()
storage/maria/ma_recovery.c:
Changed from my_getsystime() to mmicrosecond_interval_timer()
storage/maria/unittest/trnman-t.c:
Changed from my_getsystime() to mmicrosecond_interval_timer()
storage/xtradb/handler/ha_innodb.cc:
Added support for new time,datetime and timestamp
unittest/mysys/thr_template.c:
my_getsystime() -> my_interval_timer()
unittest/mysys/waiting_threads-t.c:
my_getsystime() -> my_interval_timer()
- Added a lot of code comments
- Updated get_best_ror_intersec() to prefer index scan on not clustered keys before clustered keys.
- Use HA_CLUSTERED_INDEX to define if one should use HA_MRR_INDEX_ONLY
- For test of using index or filesort to resolve ORDER BY, use HA_CLUSTERED_INDEX flag instead of primary_key_is_clustered()
- Use HA_TABLE_SCAN_ON_INDEX instead of primary_key_is_clustered() to decide if ALTER TABLE ... ORDER BY will have any effect.
sql/ha_partition.h:
Added comment with warning for code unsafe to use with multiple storage engines at the same time
sql/handler.h:
Added HA_CLUSTERED_INDEX.
Documented primary_key_is_clustered()
sql/opt_range.cc:
Added code comments
Updated get_best_ror_intersec() to ignore clustered keys.
Optimized away cpk_scan_used and one instance of current_thd (Simpler code)
Use HA_CLUSTERED_INDEX to define if one should use HA_MRR_INDEX_ONLY
sql/sql_select.cc:
Changed comment to #ifdef
For test of using index or filesort to resolve ORDER BY, use HA_CLUSTERED_INDEX flag instead of primary_key_is_clustered()
(Change is smaller than what it looks beause of indentation change)
sql/sql_table.cc:
Use HA_TABLE_SCAN_ON_INDEX instead of primary_key_is_clustered() to decide if ALTER TABLE ... ORDER BY will have any effect.
storage/innobase/handler/ha_innodb.h:
Added support for HA_CLUSTERED_INDEX
storage/innodb_plugin/handler/ha_innodb.cc:
Added support for HA_CLUSTERED_INDEX
storage/xtradb/handler/ha_innodb.cc:
Added support for HA_CLUSTERED_INDEX
Resolved all conflicts, bad merges and fixed a few minor bugs in the code.
Commented out the queries from multi_update, view, subselect_sj, func_str,
derived_view, view_grant that failed either with crashes in ps-protocol or
with wrong results.
The failures are clear indications of some bugs in the code and these bugs
are to be fixed.
