Analysis:
The fix for lp:944706 introduces early subquery optimization.
While a subquery is being optimized some of its predicates may be
removed. In the test case, the EXISTS subquery is constant, and is
evaluated to TRUE. As a result the whole OR is TRUE, and thus the
correlated condition "b = alias1.b" is optimized away. The subquery
becomes non-correlated.
The subquery cache is designed to work only for correlated subqueries.
If constant subquery optimization is disallowed, then the constant
subquery is not evaluated, the subquery remains correlated, and its
execution is cached. As a result execution is fast.
However, when the constant subquery was optimized away, it was neither
cached by the subquery cache, nor it was cached by the internal subquery
caching. The latter was due to the fact that the subquery still appeared
as correlated to the subselect_XYZ_engine::exec methods, and they
re-executed the subquery on each call to Item_subselect::exec.
Solution:
The solution is to update the correlated status of the subquery after it has
been optimized. This status consists of:
- st_select_lex::is_correlated
- Item_subselect::is_correlated
- SELECT_LEX::uncacheable
- SELECT_LEX_UNIT::uncacheable
The status is updated by st_select_lex::update_correlated_cache(), and its
caller st_select_lex::optimize_unflattened_subqueries. The solution relies
on the fact that the optimizer already called
st_select_lex::update_used_tables() for each subquery. This allows to
efficiently update the correlated status of each subquery without walking
the whole subquery tree.
Notice that his patch is an improvement over MySQL 5.6 and older, where
subqueries are not pre-optimized, and the above analysis is not possible.
sql/sql_insert.cc:
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
******
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
sql/sql_table.cc:
small cleanup
******
small cleanup
- Set the default
- Adjust the testcases so that 'new' tests are run with optimizations turned on.
- Pull out relevant tests from "irrelevant" tests and run them with optimizations on.
- Run range.test and innodb.test with both mrr=on and mrr=off
Resolved all conflicts, bad merges and fixed a few minor bugs in the code.
Commented out the queries from multi_update, view, subselect_sj, func_str,
derived_view, view_grant that failed either with crashes in ps-protocol or
with wrong results.
The failures are clear indications of some bugs in the code and these bugs
are to be fixed.
- "Using MRR" is no longer shown with range access.
- Instead, both range and BKA accesses will show one of the following:
= "Rowid-ordered scan"
= "Key-ordered scan"
= "Key-ordered Rowid-ordered scan"
depending on whether DS-MRR implementation will do scan keys in order, rowids in order,
or both.
- The patch also introduces a way for other storage engines/MRR implementations to
pass information to EXPLAIN output about the properties of employed MRR scans.
Also:
Changed the value of TIME_FOR_COMPARE_ROWID to make it the same as for MWL 21.
Changed some queries in range_vs_index_merge.test to make them generate
the same plans as earlier.
- Fixed problem with oqgraph and 'make dist'
Note that after this merge we have a problem show in join_outer where we examine too many rows in one specific case (related to BUG#57024).
This will be fixed when mwl#128 is merged into 5.3.
Open issues:
- A better fix for #57688; Igor is working on this
- Test failure in index_merge_innodb.test ; Igor promised to look at this
- Some Innodb tests fails (need to merge with latest xtradb) ; Kristian promised to look at this.
- Failing tests: innodb_plugin.innodb_bug56143 innodb_plugin.innodb_bug56632 innodb_plugin.innodb_bug56680 innodb_plugin.innodb_bug57255
- Werror is disabled; Should be enabled after merge with xtradb.
This is the 5.5 version of the fix. The 5.1 version was too complicated to
merge and was null merged.
This is a regression from the fix for bug no 38999. A storage engine capable
of reading only a subset of a table's columns updates corresponding bits in
the read buffer to signal that it has read NULL values for the corresponding
columns. It cannot, and should not, update any other bits. Bug no 38999
occurred because the implementation of UPDATE statements compare the NULL bits
using memcmp, inadvertently comparing bits that were never requested from the
storage engine. The regression was caused by the storage engine trying to
alleviate the situation by writing to all NULL bits, even those that it had no
knowledge of. This has devastating effects for the index merge algorithm,
which relies on all NULL bits, except those explicitly requested, being left
unchanged.
The fix reverts the fix for bug no 38999 in both InnoDB and InnoDB plugin and
changes the server's method of comparing records. For engines that always read
entire rows, we proceed as usual. For engines capable of reading only select
columns, the record buffers are now compared on a column by column basis. An
assertion was also added so that non comparable buffers are never read. Some
relevant copy-pasted code was also consolidated in a new function.
This is a regression from the fix for bug no 38999. A storage engine capable
of reading only a subset of a table's columns updates corresponding bits in
the read buffer to signal that it has read NULL values for the corresponding
columns. It cannot, and should not, update any other bits. Bug no 38999
occurred because the implementation of UPDATE statements compare the NULL bits
using memcmp, inadvertently comparing bits that were never requested from the
storage engine. The regression was caused by the storage engine trying to
alleviate the situation by writing to all NULL bits, even those that it had no
knowledge of. This has devastating effects for the index merge algorithm,
which relies on all NULL bits, except those explicitly requested, being left
unchanged.
The fix reverts the fix for bug no 38999 in both InnoDB and InnoDB plugin and
changes the server's method of comparing records. For engines that always read
entire rows, we proceed as usual. For engines capable of reading only select
columns, the record buffers are now compared on a column by column basis. An
assertion was also added so that non comparable buffers are never read. Some
relevant copy-pasted code was also consolidated in a new function.
Subselect executes twice, at JOIN::optimize stage
and at JOIN::execute stage. At optimize stage
Innodb prebuilt struct which is used for the
retrieval of column values is initialized in.
ha_innobase::index_read(), prebuilt->sql_stat_start is true.
After QUICK_ROR_INTERSECT_SELECT finished his job it
restores read_set/write_set bitmaps with initial values
and deactivates one of the handlers used by
QUICK_ROR_INTERSECT_SELECT in JOIN::cleanup
(it's the case when we reuse original handler as one of
handlers required by QUICK_ROR_INTERSECT_SELECT object).
On second subselect execution inactive handler is activated
in QUICK_RANGE_SELECT::reset, file->ha_index_init().
In ha_index_init Innodb prebuilt struct is reinitialized
with inappropriate read_set/write_set bitmaps. Further
reinitialization in ha_innobase::index_read() does not
happen as prebuilt->sql_stat_start is false.
It leads to partial retrieval of required field values
and we get a mix of field values from different records
in the record buffer.
The fix is to reset
read_set/write_set bitmaps as these values
are required for proper intialization of
internal InnoDB struct which is used for
the retrieval of column values
(see build_template(), ha_innodb.cc)
mysql-test/include/index_merge_ror_cpk.inc:
test case
mysql-test/r/index_merge_innodb.result:
test case
mysql-test/r/index_merge_myisam.result:
test case
sql/opt_range.cc:
if ROR merge scan is used we need to reset
read_set/write_set bitmaps as these values
are required for proper intialization of
internal InnoDB struct which is used for
the retrieval of column values
(see build_template(), ha_innodb.cc)
- Let "mysqld --help --verbose" list all optimizer options
- Make it possible to add new @@optimizer_switch flags w/o causing .result
changes all over the testsuite:
= Remove "select @@optimizer_switch" from tests that do not need all switches
= Move @@optimizer_switch-specific tests to t/optimizer_switch.test
libmysqld/Makefile.am:
The new file added.
mysql-test/r/index_merge_myisam.result:
subquery_cache optimization option added.
mysql-test/r/myisam_mrr.result:
subquery_cache optimization option added.
mysql-test/r/subquery_cache.result:
The subquery cache tests added.
mysql-test/r/subselect3.result:
Subquery cache switched off to avoid changing read statistics.
mysql-test/r/subselect3_jcl6.result:
Subquery cache switched off to avoid changing read statistics.
mysql-test/r/subselect_no_mat.result:
subquery_cache optimization option added.
mysql-test/r/subselect_no_opts.result:
subquery_cache optimization option added.
mysql-test/r/subselect_no_semijoin.result:
subquery_cache optimization option added.
mysql-test/r/subselect_sj.result:
subquery_cache optimization option added.
mysql-test/r/subselect_sj_jcl6.result:
subquery_cache optimization option added.
mysql-test/t/subquery_cache.test:
The subquery cache tests added.
mysql-test/t/subselect3.test:
Subquery cache switched off to avoid changing read statistics.
sql/CMakeLists.txt:
The new file added.
sql/Makefile.am:
The new files added.
sql/item.cc:
Expression cache item (Item_cache_wrapper) added.
Item_ref and Item_field fixed for correct usage of result field and fast resolwing in SP.
sql/item.h:
Expression cache item (Item_cache_wrapper) added.
Item_ref and Item_field fixed for correct usage of result field and fast resolwing in SP.
sql/item_cmpfunc.cc:
Subquery cache added.
sql/item_cmpfunc.h:
Subquery cache added.
sql/item_subselect.cc:
Subquery cache added.
sql/item_subselect.h:
Subquery cache added.
sql/item_sum.cc:
Registration of subquery parameters added.
sql/mysql_priv.h:
subquery_cache optimization option added.
sql/mysqld.cc:
subquery_cache optimization option added.
sql/opt_range.cc:
Fix due to subquery cache.
sql/opt_subselect.cc:
Parameters of the function cahnged.
sql/procedure.h:
.h file guard added.
sql/sql_base.cc:
Registration of subquery parameters added.
sql/sql_class.cc:
Option to allow add indeces to temporary table.
sql/sql_class.h:
Item iterators added.
Option to allow add indeces to temporary table.
sql/sql_expression_cache.cc:
Expression cache for caching subqueries added.
sql/sql_expression_cache.h:
Expression cache for caching subqueries added.
sql/sql_lex.cc:
Registration of subquery parameters added.
sql/sql_lex.h:
Registration of subqueries and subquery parameters added.
sql/sql_select.cc:
Subquery cache added.
sql/sql_select.h:
Subquery cache added.
sql/sql_union.cc:
A new parameter to the function added.
sql/sql_update.cc:
A new parameter to the function added.
sql/table.cc:
Procedures to manage temporarty tables index added.
sql/table.h:
Procedures to manage temporarty tables index added.
storage/maria/ha_maria.cc:
Fix of handler to allow destoy a table in case of error during the table creation.
storage/maria/ha_maria.h:
.h file guard added.
storage/myisam/ha_myisam.cc:
Fix of handler to allow destoy a table in case of error during the table creation.
- INSERT with RAND() doesn't require row based logging again
- Some bugs fixed in opt_range() where we table->key_read was wrongly used
.bzrignore:
Ignore new xtstat binary
mysql-test/r/index_merge_myisam.result:
Update results (old result was wrong)
mysql-test/suite/binlog/r/binlog_stm_binlog.result:
Added drop table first
mysql-test/suite/binlog/r/binlog_stm_unsafe_warning.result:
Added test for when RAND() requires row based logging
mysql-test/suite/binlog/t/binlog_stm_binlog.test:
Added drop table first
mysql-test/suite/binlog/t/binlog_stm_unsafe_warning.test:
Added test for when RAND() requires row based logging
scripts/make_binary_distribution.sh:
Removed type from last commit
sql/item_create.cc:
Don't require row based logging when using RAND() with INSERT
sql/opt_range.cc:
Revert wrong patch from Oracle:
- As QUICK_RANGE_SELECT uses it's own 'file' handler to the tables, one can't use 'table->key_read' as a flag to detect if index only read (keyread) is used or not
- Don't set keyread if keyread is already enabled
- Don't disable key read, if we didn't enable it ourselves
- Simplify code (and ensure that we do proper cleanup of index only read)
sql/opt_range.h:
Added flags to detect if the range optimizer enabled index only read (key read) or not
sql/opt_sum.cc:
Use our more optimized macros
sql/sql_lex.h:
Added 'readable' function to check if we are in a sub query function or not (not normal query or sub query in FROM clause)
sql/sql_select.cc:
Use our more optimized keyread macros
Added ASSERTS early
Simplify code on eliminate_item_equal()
Fixed that substitute_for_best_equal_field() doesn't core dump in case of out of memory conditions.
Removed not needed test for 'field->maybe_null()'
Replaced master_unit()->item with is_subquery_function() (More readable)
sql/sql_update.cc:
Use our more optimized keyread macros
sql/table.cc:
Use our more optimized keyread macros
sql/table.h:
Use separate functions to enable/disable Index only reads
- Safer, more readable, better logging and faster.
This patch does three things:
- It adds the possibility to force the execution of top-level [NOT] IN
subquery predicates via the IN=>EXISTS transformation. This is done by
setting both optimizer switches partial_match_rowid_merge and
partial_match_table_scan to "off".
- It adjusts all test cases where the complete optimizer_switch is
selected because now we have two more switches.
- For those test cases where the plan changes because of the new available
strategies, we switch off both partial match strategies in order to
force the "old" IN=>EXISTS strategy. This is done because most of these
test cases specifically test bugs in this strategy.
sql/opt_subselect.cc:
Adds the possibility to force the execution of top-level [NOT] IN
subquery predicates via the IN=>EXISTS transformation. This is done by
setting both optimizer switches partial_match_rowid_merge and
partial_match_table_scan to "off".
"set engine_condition_pushdown" is deprecated, engine condition pushdown is controlled
by a new "set optimizer_switch=engine_condition_pushdown=on|off".
mysql-test/r/index_merge_myisam.result:
@@optimizer_switch has a new flag
mysql-test/r/mysqld--help-notwin.result:
@@optimizer_switch has a new flag
mysql-test/r/mysqld--help-win.result:
@@optimizer_switch has a new flag
mysql-test/r/optimizer_switch_eng_cond_pushdown1.result:
Check how --engine-condition-pushdown and --optimizer-switch influence each other when used together (last wins).
mysql-test/r/optimizer_switch_eng_cond_pushdown2.result:
Check how --engine-condition-pushdown and --optimizer-switch influence each other when used together (last wins).
mysql-test/suite/ndb/r/ndb_condition_pushdown.result:
@@engine_condition_pushdown is deprecated, use @@optimizer_switch instead
mysql-test/suite/ndb/r/ndb_gis.result:
@@engine_condition_pushdown is deprecated, use @@optimizer_switch instead
mysql-test/suite/ndb/r/ndb_index_unique.result:
@@engine_condition_pushdown is deprecated, use @@optimizer_switch instead
mysql-test/suite/ndb/t/ndb_condition_pushdown.test:
@@engine_condition_pushdown is deprecated, use @@optimizer_switch instead
mysql-test/suite/ndb/t/ndb_gis.test:
@@engine_condition_pushdown is deprecated, use @@optimizer_switch instead
mysql-test/suite/ndb/t/ndb_index_unique.test:
@@engine_condition_pushdown is deprecated, use @@optimizer_switch instead
mysql-test/suite/sys_vars/r/engine_condition_pushdown_basic.result:
Setting @@engine_condition_pushdown gives a deprecation warning now.
We test that the engine_condition_pushdown flag of @@optimizer_switch, and @@engine_condition_pushdown
influence each other (turning the flag on/off sets the variable on/off and vice-versa).
mysql-test/suite/sys_vars/r/optimizer_switch_basic.result:
@@optimizer_switch has a new flag
mysql-test/suite/sys_vars/t/engine_condition_pushdown_basic.test:
Setting @@engine_condition_pushdown gives a deprecation warning now.
We test that the engine_condition_pushdown flag of @@optimizer_switch, and @@engine_condition_pushdown
influence each other (turning the flag on/off sets the variable on/off and vice-versa).
mysql-test/t/optimizer_switch_eng_cond_pushdown1-master.opt:
Check how --engine-condition-pushdown and --optimizer-switch influence each other when used together (last wins).
mysql-test/t/optimizer_switch_eng_cond_pushdown1.test:
Check how --engine-condition-pushdown and --optimizer-switch influence each other when used together (last wins).
mysql-test/t/optimizer_switch_eng_cond_pushdown2-master.opt:
Check how --engine-condition-pushdown and --optimizer-switch influence each other when used together (last wins).
mysql-test/t/optimizer_switch_eng_cond_pushdown2.test:
Check how --engine-condition-pushdown and --optimizer-switch influence each other when used together (last wins).
sql/mysql_priv.h:
new "engine_condition_pushdown" switch in @@optimizer_switch, on by default, like
@@engine_condition_pushdown is on by default. Constants are ULL because optimizer_switch
is stored in a ulonglong.
sql/mysqld.cc:
Making --engine-condition-pushdown and --optimizer-switch (command-line options)
influence each other (last wins)
sql/records.cc:
@@engine_condition_pushdown is deprecated, use @@optimizer_switch instead
sql/sql_select.cc:
@@engine_condition_pushdown is deprecated, use @@optimizer_switch instead
sql/sys_vars.cc:
Setting @@engine_condition_pushdown now issues a deprecation message. The version for removal
is unknown at this point so I copied it from other deprecation warnings in this file.
Turning on/off the engine_condition_pushdown flag of @@optimizer_switch (with SET) turns on/off the @@engine_condition_pushdown variable, and vice-versa, thanks to fix_* functions.
- Make index condition pushdown be controlled by an @@optimizer_switch flag,
not by @@engine_condition_pushdown
- Make MRR buffer size be controlled by @@mrr_buffer_size, not
by @@read_rnd_buffer_size
- Move parts of code to separate files
- Code cleanup
- Add --sorted_result to some SELECTs in tests.
Bug#16565 mysqld --help --verbose does not order variablesBug#20413 sql_slave_skip_counter is not shown in show variables
Bug#20415 Output of mysqld --help --verbose is incomplete
Bug#25430 variable not found in SELECT @@global.ft_max_word_len;
Bug#32902 plugin variables don't know their names
Bug#34599 MySQLD Option and Variable Reference need to be consistent in formatting!
Bug#34829 No default value for variable and setting default does not raise error
Bug#34834 ? Is accepted as a valid sql mode
Bug#34878 Few variables have default value according to documentation but error occurs
Bug#34883 ft_boolean_syntax cant be assigned from user variable to global var.
Bug#37187 `INFORMATION_SCHEMA`.`GLOBAL_VARIABLES`: inconsistent status
Bug#40988 log_output_basic.test succeeded though syntactically false.
Bug#41010 enum-style command-line options are not honoured (maria.maria-recover fails)
Bug#42103 Setting key_buffer_size to a negative value may lead to very large allocations
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
Bug#44797 plugins w/o command-line options have no disabling option in --help
Bug#46314 string system variables don't support expressions
Bug#46470 sys_vars.max_binlog_cache_size_basic_32 is broken
Bug#46586 When using the plugin interface the type "set" for options caused a crash.
Bug#47212 Crash in DBUG_PRINT in mysqltest.cc when trying to print octal number
Bug#48758 mysqltest crashes on sys_vars.collation_server_basic in gcov builds
Bug#49417 some complaints about mysqld --help --verbose output
Bug#49540 DEFAULT value of binlog_format isn't the default value
Bug#49640 ambiguous option '--skip-skip-myisam' (double skip prefix)
Bug#49644 init_connect and \0
Bug#49645 init_slave and multi-byte characters
Bug#49646 mysql --show-warnings crashes when server dies
CMakeLists.txt:
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
client/mysql.cc:
don't crash with --show-warnings when mysqld dies
config/ac-macros/plugins.m4:
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
include/my_getopt.h:
comments
include/my_pthread.h:
fix double #define
mysql-test/mysql-test-run.pl:
run sys_vars suite by default
properly recognize envirinment variables (e.g. MTR_MAX_SAVE_CORE) set to 0
escape gdb command line arguments
mysql-test/suite/sys_vars/r/rpl_init_slave_func.result:
init_slave+utf8 bug
mysql-test/suite/sys_vars/t/rpl_init_slave_func.test:
init_slave+utf8 bug
mysys/my_getopt.c:
Bug#34599 MySQLD Option and Variable Reference need to be consistent in formatting!
Bug#46586 When using the plugin interface the type "set" for options caused a crash.
Bug#49640 ambiguous option '--skip-skip-myisam' (double skip prefix)
mysys/typelib.c:
support for flagset
sql/ha_ndbcluster.cc:
backport from telco tree
sql/item_func.cc:
Bug#49644 init_connect and \0
Bug#49645 init_slave and multi-byte characters
sql/sql_builtin.cc.in:
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
sql/sql_plugin.cc:
Bug#44691 Some plugins configured as MYSQL_PLUGIN_MANDATORY in can be disabled
Bug#32902 plugin variables don't know their names
Bug#44797 plugins w/o command-line options have no disabling option in --help
sql/sys_vars.cc:
all server variables are defined here
storage/myisam/ft_parser.c:
remove unnecessary updates of param->quot
storage/myisam/ha_myisam.cc:
myisam_* variables belong here
strings/my_vsnprintf.c:
%o and %llx
unittest/mysys/my_vsnprintf-t.c:
%o and %llx tests
vio/viosocket.c:
bugfix: fix @@wait_timeout to work with socket timeouts (vs. alarm thread)
WL#2474 "Multi Range Read: Change the default MRR implementation to implement new MRR interface"
WL#2475 "Batched range read functions for MyISAM/InnoDb"
"Index condition pushdown for MyISAM/InnoDB"
Igor's fix from sp1r-igor@olga.mysql.com-20080330055902-07614:
There could be observed the following problems:
1. EXPLAIN did not mention pushdown conditions from on expressions in the
'extra' column. As a result if a query had no where conditions pushed
down to a table, but had on conditions pushed to this table the 'extra'
column in the EXPLAIN for the table missed 'using where'.
2. Conditions for ref access were not eliminated from on expressions
though such conditions were eliminated from the where condition.
