Analysis:
When the method JOIN::choose_subquery_plan() decided to apply
the IN-TO-EXISTS strategy, it set the unit and select_lex
uncacheable flag to UNCACHEABLE_DEPENDENT_INJECTED unconditionally.
As result, even if IN-TO-EXISTS injected non-correlated predicates,
the subquery was still treated as correlated.
Solution:
Set the subquery as correlated only if the injected predicate(s) depend
on the outer query.
Analysis:
The fix for lp:944706 introduces early subquery optimization.
While a subquery is being optimized some of its predicates may be
removed. In the test case, the EXISTS subquery is constant, and is
evaluated to TRUE. As a result the whole OR is TRUE, and thus the
correlated condition "b = alias1.b" is optimized away. The subquery
becomes non-correlated.
The subquery cache is designed to work only for correlated subqueries.
If constant subquery optimization is disallowed, then the constant
subquery is not evaluated, the subquery remains correlated, and its
execution is cached. As a result execution is fast.
However, when the constant subquery was optimized away, it was neither
cached by the subquery cache, nor it was cached by the internal subquery
caching. The latter was due to the fact that the subquery still appeared
as correlated to the subselect_XYZ_engine::exec methods, and they
re-executed the subquery on each call to Item_subselect::exec.
Solution:
The solution is to update the correlated status of the subquery after it has
been optimized. This status consists of:
- st_select_lex::is_correlated
- Item_subselect::is_correlated
- SELECT_LEX::uncacheable
- SELECT_LEX_UNIT::uncacheable
The status is updated by st_select_lex::update_correlated_cache(), and its
caller st_select_lex::optimize_unflattened_subqueries. The solution relies
on the fact that the optimizer already called
st_select_lex::update_used_tables() for each subquery. This allows to
efficiently update the correlated status of each subquery without walking
the whole subquery tree.
Notice that his patch is an improvement over MySQL 5.6 and older, where
subqueries are not pre-optimized, and the above analysis is not possible.
The patch enables back constant subquery execution during
query optimization after it was disabled during the development
of MWL#89 (cost-based choice of IN-TO-EXISTS vs MATERIALIZATION).
The main idea is that constant subqueries are allowed to be executed
during optimization if their execution is not expensive.
The approach is as follows:
- Constant subqueries are recursively optimized in the beginning of
JOIN::optimize of the outer query. This is done by the new method
JOIN::optimize_constant_subqueries(). This is done so that the cost
of executing these queries can be estimated.
- Optimization of the outer query proceeds normally. During this phase
the optimizer may request execution of non-expensive constant subqueries.
Each place where the optimizer may potentially execute an expensive
expression is guarded with the predicate Item::is_expensive().
- The implementation of Item_subselect::is_expensive has been extended
to use the number of examined rows (estimated by the optimizer) as a
way to determine whether the subquery is expensive or not.
- The new system variable "expensive_subquery_limit" controls how many
examined rows are considered to be not expensive. The default is 100.
In addition, multiple changes were needed to make this solution work
in the light of the changes made by MWL#89. These changes were needed
to fix various crashes and wrong results, and legacy bugs discovered
during development.
The not_null_tables() of Item_func_not_all and Item_in_optimizer was inherited from
Item_func by mistake. It made the optimizer think that subquery
predicates with ALL/ANY/IN were null-rejecting. This could trigger invalid
conversions of outer joins into inner joins.
mysql-test/suite/innodb/t/group_commit_crash.test:
remove autoincrement to avoid rbr being used for insert ... select
mysql-test/suite/innodb/t/group_commit_crash_no_optimize_thread.test:
remove autoincrement to avoid rbr being used for insert ... select
mysys/my_addr_resolve.c:
a pointer to a buffer is returned to the caller -> the buffer cannot be on the stack
mysys/stacktrace.c:
my_vsnprintf() is ok here, in 5.5
This bug appeared after the patch for bug 939009 that in the
function merge_key_fields forgot to reset a proper value for
the val field in the result of the merge operation of the key
field created for a regular key access and the key field
created to look for a NULL key.
Adjusted the results of the test case for bug 939009 that
actually were incorrect.
IS EXECUTED TWICE FROM P
This bug is a duplicate of bug 12567331, which was pushed to the
optimizer backporting tree on 2011-06-11. This is just a back-port of
the fix. Both test cases are included as they differ somewhat.
sql/sql_insert.cc:
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
******
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
sql/sql_table.cc:
small cleanup
******
small cleanup
of the 5.3 code line after a merge with 5.2 on 2010-10-28
in order not to allow the cost to access a joined table to be equal
to 0 ever.
Expanded data sets for many test cases to get the same execution plans
as before.
For any query JOIN::optimize() should call the method
SELECT::save_leaf_tables after the last transformation
that utilizes the statement memory rather than the
execution memory.
This bug is a special case of lp:813447.
Analysis:
Constant optimization finds that the condition t2.a = 1
can be used to access the primary key of table 't2'. As
a result both outer table t1,t2 are considered as constant
when we reach the execution phase. At the same time, during
constant optimization, the IN predicate is not evaluated
because it is expensive.
When execution of the outer query reaches do_select(),
control flow enter the branch:
if (join->table_count == join->const_tables)
{ ... }
This branch checks only the WHERE and HAVING clauses,
but doesn't check the ON clauses of the query. Since the
IN predicate was not evaluated during optimization, it is
not evaluated at all, thus execution doesn't detect that
the ON clause is FALSE.
Solution:
Similar to the patch for bug lp:813447, exclude system
tables from constant substitution based on unique key
lookups if there is an expensive ON condition on the
inner table.
A lot of small fixes and new test cases.
client/mysqlbinlog.cc:
Cast removed
client/mysqltest.cc:
Added missing DBUG_RETURN
include/my_pthread.h:
set_timespec_time_nsec() now only takes one argument
mysql-test/t/date_formats.test:
Remove --disable_ps_protocl as now also ps supports microseconds
mysys/my_uuid.c:
Changed to use my_interval_timer() instead of my_getsystime()
mysys/waiting_threads.c:
Changed to use my_hrtime()
sql/field.h:
Added bool special_const_compare() for fields that may convert values before compare (like year)
sql/field_conv.cc:
Added test to get optimal copying of identical temporal values.
sql/item.cc:
Return that item_int is equal if it's positive, even if unsigned flag is different.
Fixed Item_cache_str::save_in_field() to have identical null check as other similar functions
Added proper NULL check to Item_cache_int::save_in_field()
sql/item_cmpfunc.cc:
Don't call convert_constant_item() if there is nothing that is worth converting.
Simplified test when years should be converted
sql/item_sum.cc:
Mark cache values in Item_sum_hybrid as not constants to ensure they are not replaced by other cache values in compare_datetime()
sql/item_timefunc.cc:
Changed sec_to_time() to take a my_decimal argument to ensure we don't loose any sub seconds.
Added Item_temporal_func::get_time() (This simplifies some things)
sql/mysql_priv.h:
Added Lazy_string_decimal()
sql/mysqld.cc:
Added my_decimal constants max_seconds_for_time_type, time_second_part_factor
sql/table.cc:
Changed expr_arena to be of type CONVENTIONAL_EXECUTION to ensure that we don't loose any items that are created by fix_fields()
sql/tztime.cc:
TIME_to_gmt_sec() now sets *in_dst_time_gap in case of errors
This is needed to be able to detect if timestamp is 0
storage/maria/lockman.c:
Changed from my_getsystime() to set_timespec_time_nsec()
storage/maria/ma_loghandler.c:
Changed from my_getsystime() to my_hrtime()
storage/maria/ma_recovery.c:
Changed from my_getsystime() to mmicrosecond_interval_timer()
storage/maria/unittest/trnman-t.c:
Changed from my_getsystime() to mmicrosecond_interval_timer()
storage/xtradb/handler/ha_innodb.cc:
Added support for new time,datetime and timestamp
unittest/mysys/thr_template.c:
my_getsystime() -> my_interval_timer()
unittest/mysys/waiting_threads-t.c:
my_getsystime() -> my_interval_timer()
and collateral changes.
* introduce my_hrtime_t, my_timediff_t, and conversion macros
* inroduce TIME_RESULT, but it can only be returned from Item::cmp_type(),
never from Item::result_type()
* pack_time/unpack_time function for "packed" representation of
MYSQL_TIME in a longlong that can be compared
* ADDTIME()/SUBTIME()/+- INTERVAL now work with TIME values
* numbers aren't quoted in EXPLAIN EXTENDED
* new column I_S.COLUMNS.DATETIME_PRECISION
* date/time values are compares to anything as date/time, not as strings or numbers.
* old timestamp(X) is no longer supported
* MYSQL_TIME to string conversion functions take precision as an argument
* unified the warnings from Field_timestamp/datetime/time/date/newdate store methods
* Field_timestamp_hires, Field_datetime_hires, Field_time_hires
* Field_temporal
* Lazy_string class to pass a value (string, number, time) polymorphically down the stack
* make_truncated_value_warning and Field::set_datetime_warning use Lazy_string as an argument, removed char*/int/double variants
* removed Field::can_be_compared_as_longlong(). Use Field::cmp_type() == INT_RESULT instead
* introduced Item::cmp_result() instead of Item::is_datetime() and Item::result_as_longlong()
* in many cases date/time types are treated like other types, not as special cases
* greatly simplified Arg_comparator (regarding date/time/year code)
* SEC_TO_TIME is real function, not integer.
* microsecond precision in NOW, CURTIME, etc
* Item_temporal. All items derived from it only provide get_date, but no val* methods
* replication of NOW(6)
* Protocol::store(time) now takes the precision as an argument
* @@TIMESTAMP is a double
client/mysqlbinlog.cc:
remove unneded casts
include/my_sys.h:
introduce my_hrtime_t, my_timediff_t, and conversion macros
include/my_time.h:
pack_time/unpack_time, etc.
convenience functions to work with MYSQL_TIME::second_part
libmysql/libmysql.c:
str_to_time() is gone. str_to_datetime() does it now.
my_TIME_to_str() takes the precision as an argument
mysql-test/include/ps_conv.inc:
time is not equal to datetime anymore
mysql-test/r/distinct.result:
a test for an old MySQL bug
mysql-test/r/explain.result:
numbers aren't quoted in EXPLAIN EXTENDED
mysql-test/r/func_default.result:
numbers aren't quoted in EXPLAIN EXTENDED
mysql-test/r/func_sapdb.result:
when decimals=NOT_FIXED_DEC it means "not fixed" indeed
mysql-test/r/func_test.result:
numbers aren't quoted in EXPLAIN EXTENDED
mysql-test/r/func_time.result:
ADDTIME()/SUBTIME()/+- INTERVAL now work with TIME values
mysql-test/r/having.result:
numbers aren't quoted in EXPLAIN EXTENDED
mysql-test/r/information_schema.result:
new column I_S.COLUMNS.DATETIME_PRECISION
mysql-test/r/join_outer.result:
numbers aren't quoted in EXPLAIN EXTENDED
mysql-test/r/metadata.result:
TIMESTAMP no longer has zerofill flag
mysql-test/r/range.result:
invalid datetime is not compared with as a string
mysql-test/r/select.result:
NO_ZERO_IN_DATE, etc only affect storage - according to the manual
numbers aren't quoted in EXPLAIN EXTENDED
mysql-test/r/subselect.result:
numbers aren't quoted in EXPLAIN EXTENDED
mysql-test/r/sysdate_is_now.result:
when decimals=NOT_FIXED_DEC it means "not fixed" indeed
mysql-test/r/type_blob.result:
TIMESTAMP(N) is not deprecated
mysql-test/r/type_timestamp.result:
old TIMESTAMP(X) semantics is not supported anymore
mysql-test/r/union.result:
numbers aren't quoted in EXPLAIN EXTENDED
mysql-test/r/varbinary.result:
numbers aren't quoted in EXPLAIN EXTENDED
mysql-test/t/distinct.test:
test for an old MySQL bug
mysql-test/t/func_time.test:
+- INTERVAL now works with TIME values
mysql-test/t/select.test:
typo
mysql-test/t/subselect.test:
only one error per statement, please
mysql-test/t/system_mysql_db_fix40123.test:
old timestamp(X) is no longer supported
mysql-test/t/system_mysql_db_fix50030.test:
old timestamp(X) is no longer supported
mysql-test/t/system_mysql_db_fix50117.test:
old timestamp(X) is no longer supported
mysql-test/t/type_blob.test:
old timestamp(X) is no longer supported
mysql-test/t/type_timestamp.test:
old timestamp(X) is no longer supported
mysys/my_getsystime.c:
functions to get the time with microsecond precision
mysys/my_init.c:
move the my_getsystime.c initialization code to my_getsystime.c
mysys/my_static.c:
no need to make these variables extern
mysys/my_static.h:
no need to make these variables extern
scripts/mysql_system_tables.sql:
old timestamp(X) is no longer supported
scripts/mysql_system_tables_fix.sql:
old timestamp(X) is no longer supported
scripts/mysqlhotcopy.sh:
old timestamp(X) is no longer supported
sql-common/my_time.c:
* call str_to_time from str_to_datetime, as appropriate
* date/time to string conversions take precision as an argument
* number_to_time()
* TIME_to_double()
* pack_time() and unpack_time()
sql/event_data_objects.cc:
cast is not needed
my_datetime_to_str() takes precision as an argument
sql/event_db_repository.cc:
avoid dangerous downcast (because the pointer is
not always Field_timestamp, see events_1.test)
sql/event_queue.cc:
avoid silly double-work for cond_wait
(having an endpoint of wait, subtract the current time to get the timeout,
and use set_timespec() macro to fill in struct timespec, by adding the current
time to the timeout)
sql/field.cc:
* remove virtual Field::get_time(), everyone should use only Field::get_date()
* remove lots of #ifdef WORDS_BIGENDIAN
* unified the warnings from Field_timestamp/datetime/time/date/newdate store methods
* Field_timestamp_hires, Field_datetime_hires, Field_time_hires
* Field_temporal
* make_truncated_value_warning and Field::set_datetime_warning use Lazy_string as an argument, removed char*/int/double variants
sql/field.h:
* remove virtual Field::get_time(), everyone should use only Field::get_date()
* remove lots of #ifdef WORDS_BIGENDIAN
* unified the warnings from Field_timestamp/datetime/time/date/newdate store methods
* Field_timestamp_hires, Field_datetime_hires, Field_time_hires
* Field_temporal
* make_truncated_value_warning and Field::set_datetime_warning use Lazy_string as an argument, removed char*/int/double variants
* removed Field::can_be_compared_as_longlong(). Use Field::cmp_type() == INT_RESULT instead
sql/filesort.cc:
TIME_RESULT, cmp_time()
sql/item.cc:
* numbers aren't quoted in EXPLAIN EXTENDED
* Item::cmp_result() instead of Item::is_datetime() and Item::result_as_longlong()
* virtual Item::get_time() is gone
* Item_param::field_type() is set correctly
* Item_datetime, for a datetime constant
* time to anything is compared as a time
* Item_cache::print() prints the value is available
* bug fixed in Item_cache_int::val_str()
sql/item.h:
* Item::print_value(), to be used from Item_xxx::print() when needed
* Item::cmp_result() instead of Item::is_datetime() and Item::result_as_longlong()
* virtual Item::get_time() is gone
* Item_datetime, for a datetime constant
* better default for cast_to_int_type()
* Item_cache objects now *always* have the field_type() set
sql/item_cmpfunc.cc:
* get_year_value, get_time_value are gone. get_datetime_value does it all
* get_value_a_func, get_value_b_func are gone
* can_compare_as_dates() is gone too, TIME_RESULT is used instead
* cmp_type() instead or result_type() when doing a comparison
* compare_datetime and compate_e_datetime in the comparator_matrix, is_nulls_eq is gone
* Item::cmp_result() instead of Item::is_datetime() and Item::result_as_longlong()
sql/item_cmpfunc.h:
greatly simplified Arg_comparator
sql/item_create.cc:
* fix a bug in error messages in CAST
sql/item_func.cc:
Item::cmp_result() instead of Item::is_datetime() and Item::result_as_longlong()
mention all possibitiles in switch over Item_result values, or use default:
sql/item_row.h:
overwrite the default cmp_type() for Item_row,
as no MYSQL_TYPE_xxx value corresponds to ROW_RESULT
sql/item_timefunc.cc:
rewrite make_datetime to support precision argument
SEC_TO_TIME is real function, not integer.
many functions that returned temporal values had duplicate code in val_* methods,
some of them did not have get_date() which resulted in unnecessary date->str->date conversions.
Now they all are derived from Item_temporal_func and *only* provide get_date, not val* methods.
many fixes to set decimals (datetime precision) correctly.
sql/item_timefunc.h:
SEC_TO_TIME is real function, not integer.
many functions that returned temporal values had duplicate code in val_* methods,
some of them did not have get_date() which resulted in unnecessary date->str->date conversions.
Now they all are derived from Item_temporal_func and *only* provide get_date, not val* methods.
many fixes to set decimals (datetime precision) correctly.
sql/log_event.cc:
replication of NOW(6)
sql/log_event.h:
replication of NOW(6)
sql/mysql_priv.h:
Lazy_string class to pass a value (string, number, time) polymorphically down the stack.
make_truncated_value_warning() that uses it.
sql/mysqld.cc:
datetime in Arg_comparator::comparator_matrix
sql/opt_range.cc:
cleanup: don't disable warnings before calling save_in_field_no_warnings()
sql/protocol.cc:
Protocol::store(time) now takes the precision as an argument
sql/protocol.h:
Protocol::store(time) now takes the precision as an argument
sql/rpl_rli.cc:
small cleanup
sql/set_var.cc:
SET TIMESTAMP=double
sql/set_var.h:
@@TIMESTAMP is a double
sql/share/errmsg.txt:
precision and scale are unsigned
sql/slave.cc:
replication of NOW(6)
sql/sp_head.cc:
cleanup
sql/sql_class.cc:
support for NOW(6)
sql/sql_class.h:
support for NOW(6)
sql/sql_insert.cc:
support for NOW(6)
sql/sql_select.cc:
use item->cmp_type().
move a comment where it belongs
sql/sql_show.cc:
new column I_S.COLUMNS.DATETIME_PRECISION
sql/sql_yacc.yy:
TIME(X), DATETIME(X), cast, NOW(X), CURTIME(X), etc
sql/time.cc:
fix date_add_interval() to support MYSQL_TIMESTAMP_TIME argument
storage/myisam/ha_myisam.cc:
TIMESTAMP no longer carries ZEROFIELD flag, still we keep MYI file compatible.
strings/my_vsnprintf.c:
warnings
tests/mysql_client_test.c:
old timestamp(X) does not work anymore
datetime is no longer equal to time
in combination with IS NULL'
As this bug is a duplicate of bug#49322, it also includes test cases
covering this bugreport
Qualifying an OUTER JOIN with the condition 'WHERE <column> IS NULL',
where <column> is declared as 'NOT NULL' causes the
'not_exists_optimize' to be enabled by the optimizer.
In evaluate_join_record() the 'not_exists_optimize' caused
'NESTED_LOOP_NO_MORE_ROWS' to be returned immediately
when a matching row was found.
However, as the 'not_exists_optimize' is derived from
'JOIN_TAB::select_cond', the usual rules for condition guards
also applies for 'not_exist_optimize'. It is therefore incorrect
to check 'not_exists_optimize' without ensuring that all guards
protecting it is 'open'.
This fix uses the fact that 'not_exists_optimize' is derived from
a 'is_null' predicate term in 'tab->select_cond'. Furthermore,
'is_null' will evaluate to 'false' for any 'non-null' rows
once all guards protecting the is_null is open.
We can use this knowledge as an implicit guard check for the
'not_exists_optimize' by moving 'if (...not_exists_optimize)'
inside the handling of 'select_cond==false'. It will then
not take effect before its guards are open.
We also add an assert which requires that a
'not_exists_optimize' always comes together with
a select_cond. (containing 'is_null').
Item_equal::val_int() checked for NULL-values by checking Item::null_value
*before* the respective ::store_value() and ::cmp(Item*) metods where called.
As Item::null_value is set by these metods, the value of 'null_value'
is not valid until *after* ::store_value() or ::cmp() has
been called for the Item object.
Fix is to swap order of ::store_value()/::cmp() and checking of Item::null_value.
This pattern is widely used other places inside item_cmpfunc.cc .
- Fixed problem with oqgraph and 'make dist'
Note that after this merge we have a problem show in join_outer where we examine too many rows in one specific case (related to BUG#57024).
This will be fixed when mwl#128 is merged into 5.3.
Open issues:
- A better fix for #57688; Igor is working on this
- Test failure in index_merge_innodb.test ; Igor promised to look at this
- Some Innodb tests fails (need to merge with latest xtradb) ; Kristian promised to look at this.
- Failing tests: innodb_plugin.innodb_bug56143 innodb_plugin.innodb_bug56632 innodb_plugin.innodb_bug56680 innodb_plugin.innodb_bug57255
- Werror is disabled; Should be enabled after merge with xtradb.
Lines below which were added in the patch for Bug#56814 cause this crash:
+ if (table->table)
+ table->table->maybe_null= FALSE;
Consider following test case:
--
CREATE TABLE t1(f1 INT NOT NULL);
INSERT INTO t1 VALUES (16777214),(0);
SELECT COUNT(*) FROM t1 LEFT JOIN t1 t2
ON 1 WHERE t2.f1 > 1 GROUP BY t2.f1;
DROP TABLE t1;
--
We set TABLE::maybe_null to FALSE for t2 table
and in create_tmp_field() we create appropriate tmp table field
using create_tmp_field_from_item() function instead of
create_tmp_field_from_field. As a result we have
LONGLONG field. As we have GROUP BY clause we calculate
group buffer length, see calc_group_buffer().
Item from group list which is used for calculation
refer to the field from real tables and have LONG type.
So group buffer length become insufficient for storing of
LONGLONG value. It leads to overwriting of wrong memory
area in do_field_int() function which is called from
end_update().
After some investigation I found out that
create_tmp_field_from_item() is used only for OLAP
grouping and can not be used for common grouping
as it could be an incompatibility between tmp
table fields and group buffer length.
We can not remove create_tmp_field_from_item() call from
create_tmp_field as OLAP needs it and we can not use this
function for common grouping. So we should remove setting
TABLE::maybe_null to FALSE from simplify_joins().
In this case we'll get wrong behaviour of
list_contains_unique_index() back. To fix it we
could use Field::real_maybe_null() check instead of
Field::maybe_null() and add addition check of
TABLE_LIST::outer_join.
mysql-test/r/group_by.result:
test case
mysql-test/r/join_outer.result:
test case
mysql-test/t/group_by.test:
test case
mysql-test/t/join_outer.test:
test case
sql/sql_select.cc:
--remove wrong code
--use Field::real_maybe_null() check instead of
Field::maybe_null() and add addition check of
TABLE_LIST::outer_join
Employed the same kind of optimization as in the fix for the cases
when join buffer is used.
The optimization performs early evaluation of the conditions from
on expression with table references to only outer tables of
an outer join.
Applied the fix for bug #47217 from the mysql-6.0 codebase.
The patch adds not null predicates generated for the left parts
of the equality predicates used for ref accesses. This is done
for such predicates both in where conditions and on conditions.
For the where conditions the not null predicates were generated
but in 5.0/5.1 they actually never were used due to some lame
merge from 4.1 to 5.0. The fix for bug #47217 made these
predicates to be used in the condition pushed to the tables.
Yet only this patch generates not null predicates for equality
predicated from on conditions of outer joins.
This patch introduces a performance regression that can be
observed on a test case from null_key.test. The regression
will disappear after the fix for bug #57024 from mariadb-5.1
is pulled into mariadb-5.3.
The patch contains many changes in the outputs of the EXPLAIN
commands since generated not null predicates are considered as
parts of the conditions pushed to join tables and may add
'Usingwhere' in some rows of EXPLAINs where there used
to be no such comments.
The condition over the outer tables now are extracted from
the on condition of any outer join. This condition is
saved in a special field of the JOIN_TAB structure for
the first inner table of the outer join. The condition
is checked before the first inner table is accessed. If
it turns out to be false the table is not accessed at all
and a null complemented row is generated immediately.
