Analysis:
A query with implicit grouping is one with aggregate functions and
no GROUP BY clause. MariaDB inherits from MySQL an SQL extenstion
that allows mixing aggregate functions with non-aggregate fields.
If a query with such mixed select clause produces an empty result
set, the meaning of aggregate functions is well defined - either
NULL (MIN, MAX, etc.), or 0 (count(*)). However the non-aggregated
fields must also have some value, and the only reasonable value in
the case of empty result is NULL.
The cause of the many wrong results was that if a field is declared
as non-nullable (e.g. because it is a PK or NOT NULL), the semantic
analysis and the optimization phases treat this field as non-nullable,
and generate all related query plan elements based on this assumption.
Later during execution, these incorrectly configured/generated query
plan elements result in a wrong result because the selected fields
are not null due to the not-null assumption during optimization.
Solution:
Detect before the context analysys phase that a query uses implicit
grouping with mixed aggregates/non-aggregates, and set all fields
as nullable. The parser already walks the SELECT clause, and
already sets Item::with_sum_func for Items that reference aggreagate
functions. The patch adds a symmetric Item::with_field so that all
Items that reference an Item_field are marked during their
construction at parse time in the same way as with aggregate function
use.
Changed test suite to use --log-basename (to get the code tested)
Added --sync-sys=1 to test suite to speed it up.
Better error messages if something goes wrong with mysql_install_db
mysql-test/Makefile.am:
Removed not existing directory
mysql-test/lib/My/ConfigFactory.pm:
Use log-basename
We had to also set 'log_error' as some test was explicitely using the old name
Added 'sync-sys=1' to speed up test suite
mysql-test/r/variables-notembedded.result:
Updated test results (variable relay_log is now set)
mysql-test/suite/binlog/t/binlog_delete_and_flush_index-master.opt:
Force specific names for some log files.
mysql-test/suite/binlog/t/binlog_index-master.opt:
Force specific names for some log files.
mysql-test/suite/binlog/t/binlog_stm_unsafe_warning-master.opt:
Force specific names for some log files.
mysql-test/suite/binlog/t/binlog_stm_unsafe_warning.test:
Better error message if something goes wrong
mysql-test/suite/rpl/r/rpl_flushlog_loop.result:
Updated results
mysql-test/suite/rpl/rpl_1slave_base.cnf:
Use --log-basename
scripts/mysql_install_db.sh:
More information to --help
Write url to knowledge base if something goes wrong
Fail at once if we can't create a database directory (no reason to continue and write a screenful of not related text)
scripts/mysqld_safe.sh:
Also allow one to use --data for --datadir (common shortening)
Added support for --log-basename
Fail at once if we can't create a log directory
Fixed bug where we used a pid file name without '.pid' extension
sql/log.cc:
Create a log file name trough my_once_alloc() (To get it automaticly freed at exit)
sql/mysql_priv.h:
Added new prototype
sql/mysqld.cc:
Added support for --log-basename
Better help for a lot of log-filename related variables.
sql/rpl_rli.cc:
Write information that one can use --log-basename
sql/set_var.cc:
Add log_basename as a readonly variable
Analysis (BUG#719198):
The assert failed because the execution code for
partial matching is designed with the assumption that
NULLs on the left side are detected as early as possible,
and a NULL result is returned before any lookups are
performed at all.
However, in the case of an Item_cache object on the left
side, null was not detected properly, because detection
was done via Item::is_null(), which is not implemented at
all for Item_cache, and resolved to the default Item::is_null()
which always returns FALSE.
Solution:
Imlpement Item::is_null().
******
Analysis (BUG#730604):
The method Item_field::is_null() determines if an item is NULL from its
Item_field::field object. However, for Item_fields that represent internal
temporary tables, Item_field::field represents the field of the original
table that was the source for the temporary table (in this case t1.f3).
Both in the committed test case, and in the original bug report the current
value of t1.f3 is not NULL. This results in an incorrect count of NULLs
for this column. As a consequence, all related Ordered_key buffers are
allocated with incorrect sizes. Depending on the exact query and data,
these incorrect sizes result in various crashes or failed asserts.
Solution:
The correct value of the current field of the internal temp table is
in Item_field::result_field. This value is determined by
Item::is_null_result().
Analysis:
The assert failed because the execution code for
partial matching is designed with the assumption that
NULLs on the left side are detected as early as possible,
and a NULL result is returned before any lookups are
performed at all.
However, in the case of an Item_cache object on the left
side, null was not detected properly, because detection
was done via Item::is_null(), which is not implemented at
all for Item_cache, and resolved to the default Item::is_null()
which always returns FALSE.
Solution:
Use the property Item::null_value instead of is_null(), which
is properly updated for Item_cache objects as well.
If join condition is of the form <t2.key>=<t1.no_key> then the server
performs no index look-ups when looking for matching rows of t2 for
the rows from t1 with t1.no_key=NULL. It happens because the function
add_not_null_conds() injects an additional condition of the form
IS NOT NULL(<t1.no_key>) into the WHERE condition.
However if the join condition was of the form <t.key>=<outer_ref> no
additional null rejecting predicate was generated. This could lead
to extra records in the result set if the value of <outer_ref> happened
to be NULL.
The new code injects null rejecting predicates of the form
IS NOT NULL(<outer_ref>) and evaluates them before the first row
the subquery is constructed.
- put the code that sets HA_NULL_PART bit back
- Fix test_if_ref/part_of_refkey() so that
= NULL-ability of lookup columns does not prevent the equality
from being removed (we now have early/late NULLs filtering which
will filter out NULL values)
= equality is not removed if it is ref_or_null access, and the value
of the lookup column can alternate between the lookup value and NULL.
- The problem was that Mrr_ordered_index_reader's interrupt_read() and resume_read() would
save and restore 1) index tuple 2) the rowid (as bytes returned by handler->position()). Clustered
primary key columns were not saved/restored.
They are not explicitly present in the index tuple (i.e. table->key_info[secondary_key].key_parts
doesn't list them), but they are actually there, in particular
table->field[clustered_primary_key_member].part_of_key(secondary_key) == 1. Index condition pushdown
code [correctly] uses the latter as inidication that pushed index condition can refer to clustered PK
members.
The fix was to make interrupt_read()/resume_read() to save/restore clustered primary key members as well,
so that we get correct values for them when evaluating pushed index condition.
[3rd attempt: remove the debugging aids, fix comments in testcase]
Analysis:
The reason for the crash was that the inner subquery was executed
via a scan on a final temporary table applied after all other
operations. This final operation is implemented by changing the
contents of the JOIN object of the subquery to represent a table
scan over the temp table. At the same time query optimization of
the outer subquery required evaluation of the inner subquery, which
happened before the actual EXPLAIN. The evaluation left the JOIN
object of the inner subquery in the changed state, where it represented
a table scan over a temp table, and EXPLAIN crashed because the temp
table is not associated with any table reference (TABLE_LIST object).
The reason the JOIN was not restored was because its saving/restoration
was controlled by the join->select_lex->uncacheable flag, which was
not set in the case of materialization.
Solution:
In the methods Item_in_subselect::[single | row]_value_transformer() set:
select_lex->uncacheable|= UNCACHEABLE_EXPLAIN;
In addition, for symmetry, change:
master_unit->uncacheable|= UNCACHEABLE_EXPLAIN;
instead of UNCACHEABLE_DEPENDENT because if a subquery was not
dependent initially, the changed methods do not change this
fact. The subquery may later become correlated if it is transformed
to an EXISTS query, but it may stay uncorrelated if executed via
materialization.
- Make DsMrr_impl::dsmrr_init() handle the case of
1. 1st MRR scan using DS-MRR strategy (i.e. doing key sorting and rowid sorting)
2. 2nd MRR scan getting a buffer that's too small to fit one key element
and one rowid element, and so falling back to default MRR implementation
In this case, dsmrr_init() is invoked with {primary_handler, secondary_handler}
initialized for DS-MRR scan and have to reset them to be initialized for the
default MRR scan.
(attempt 2, with simplified testcase)
- Make equality-substitution-for-ref-access code in JOIN::optimize() treat join_tab->ref.key_copy correctly
(in the way create_ref_for_key() has filled it).
The bug in the function print_keyuse() caused crashes if
hash join could be used. It happened because the function
ignored the fact that KEYUSE structures could be created
for hash joins as well.
- Make get_constant_key_infix() take into account that there may be SEL_TREEs with
type=SEL_ARG::MAYBE_KEY, which it cannot process, because they are not real ranges
but rather indications that we might have been able to construct a range if we had
values for some other tables' fields.
(check_quick_select() already has such check)
even in the cases when there existed range/index-merge scans that
were cheaper than the full table scan.
This was a defect/bug of the implementation of mwl #128.
Now hash join can work not only with full table scan of the joined
table, but also with full index scan, range and index-merge scans.
Accordingly, in the cases when hash join is used the column 'type'
in the EXPLAINs can contain now 'hash_ALL', 'hash_index', 'hash_range'
and 'hash_index_merge'. If hash join is coupled with a range/index_merge
scan then the columns 'key' and 'key_len' contain info not only on
the used hash index, but also on the indexes used for the scan.
- Linking now with g++ instead of gcc with 'compile-dist' to solve problems with handlersocket/client
- Fixed bug in heap tables when doing handler read next-prev over last row
BUILD/compile-dist:
- Linking now with g++ instead of gcc with 'compile-dist' to solve problems with handlersocket/client
cmd-line-utils/libedit/vi.c:
Fixed compiler warning about not checking return value for write
mysql-test/r/index_intersect.result:
Updated results (missed this file in my last push)
mysql-test/suite/handler/aria.result:
Updated test results
mysql-test/suite/handler/handler.inc:
Changed test to use read next/read prev on key where there are duplicates that can come in different order depending on system
Added testing of read next-prev over last row and read prev-next around first row
mysql-test/suite/handler/heap.result:
Updated test results
mysql-test/suite/handler/init.inc:
More rows to test
mysql-test/suite/handler/innodb.result:
Updated test results
mysql-test/suite/handler/interface.result:
Updated test results
mysql-test/suite/handler/myisam.result:
Updated test results
mysql-test/t/variables-big.test:
Fixed test to not fail on windows
mysql-test/valgrind.supp:
Removed not matching fun: to get rid of valgrind warning
storage/heap/hp_rfirst.c:
Added state so that we know if we have an active position in the index.
storage/heap/hp_rkey.c:
Added state so that we know if we have an active position in the index.
storage/heap/hp_rnext.c:
Handle reading several next after finding the last row (this caused a crash before)
storage/heap/hp_rprev.c:
Handle reading several prev after finding the first row (this caused a crash before)
storage/xtradb/buf/buf0buf.c:
Fixed compiler warning about uninitialized value
dbug/dbug.c:
Fixed compiler warnings on windows
mysql-test/r/index_intersect_innodb.result:
Don't print number of rows as this varies
mysql-test/suite/funcs_1/r/processlist_val_no_prot.result:
Update to use new State
mysql-test/suite/handler/heap.result:
Update results
mysql-test/suite/handler/heap.test:
Changed key used in test as the key 'b' will not guarantee order of the two last elements (as the key value is the same)
mysql-test/suite/pbxt/r/lock_multi.result:
Update to use new State
mysql-test/t/index_intersect.test:
Don't print number of rows as this varies
mysql-test/valgrind.supp:
Added suppression for linux
mysys/hash.c:
Fixed compiler warnings on windows
plugin/handler_socket/handlersocket/database.cpp:
Fixed compiler warnings
sql-common/client_plugin.c:
Changed variable to define to avoid compiler warnings when variable is not used
sql-common/my_time.c:
Initialize all my_time elements to avoid compiler warnings
storage/oqgraph/ha_oqgraph.cc:
Use right function (to avoid compiler warning)
storage/pbxt/src/table_xt.cc:
Initialize variables (to avoid compiler warnings)
Split the tests for MWL#89 into two parts - one for bugs
(currently active), and one for functionality tets
(currently in progress, and thus disabled).
Disable the test for LP BUG#718593.
- Fixed some issues with partitions and connection_string, which also fixed lp:716890 "Pre- and post-recovery crash in Aria"
- Fixed wrong assert in Aria
Now need to merge with latest xtradb before pushing
sql/ha_partition.cc:
Ensure that m_ordered_rec_buffer is not freed before close.
sql/mysqld.cc:
Changed to use opt_stack_trace instead of opt_pstack.
Removed references to pstack
sql/partition_element.h:
Ensure that connect_string is initialized
storage/maria/ma_key_recover.c:
Fixed wrong assert
sql/multi_range_read.cc:
Added printing of error if something goes wrong in get_next()
(Not critical for this bug fix, but this was something that I noticed while testing and found missing)
storage/myisam/mi_rkey.c:
Fixed wrong error number in mi_yield_and_check_if_killed()
- In Maria/MyISAM: Release/re-acquire locks to give queries that wait on them a chance to make progress
- In Maria/MyISAM: Change from numeric constants to ICP_RES values.
- In Maria: Do check index condition in maria_rprev() (was lost in the merge/backport?)
- In Maria/MyISAM/XtraDB: Check if the query was killed, and return immediately if it was.
Added new storage engine error: HA_ERR_ABORTED_BY_USER, for handler to signal that it detected a kill of the query and aborted
Authors: Sergey Petrunia & Monty
include/my_base.h:
Added HA_ERR_ABORTED_BY_USER, for handler to signal that it detected a kill of the query and aborted
include/my_handler.h:
Added comment
mysql-test/r/myisam_icp.result:
Updated test
mysql-test/t/myisam_icp.test:
Drop used tables at start of test
Added test case that can help with manual testing of killing index condition pushdown query.
mysys/my_handler_errors.h:
Text for new storage engine error
sql/handler.cc:
If engine got HA_ERR_ABORTED_BY_USER, send kill message.
sql/multi_range_read.cc:
Return error code
storage/maria/ha_maria.cc:
Added ma_killed_in_mariadb() to detect kill.
Ensure that file->external_ref points to TABLE object.
storage/maria/ma_extra.c:
Dummy test-if-killed for standalone
storage/maria/ma_key.c:
If ma_check_index_cond() fails, set my_errno and info->cur_row.lastpos
storage/maria/ma_rkey.c:
Release/re-acquire locks to give queries that wait on them a chance to make progress
Check if the query was killed, and return immediately if it was
storage/maria/ma_rnext.c:
Check if the query was killed, and return immediately if it was
Added missing fast_ma_writeinfo(info)
storage/maria/ma_rnext_same.c:
Check if the query was killed, and return immediately if it was
Added missing fast_ma_writeinfo(info)
storage/maria/ma_rprev.c:
Check if the query was killed, and return immediately if it was
Added missing fast_ma_writeinfo(info) and ma_check_index_cond()
storage/maria/ma_search.c:
Give error message if we find a wrong key
storage/maria/ma_static.c:
Added pointer to test-if-killed function
storage/maria/maria_def.h:
New prototypes
storage/myisam/ha_myisam.cc:
Added mi_killed_in_mariadb()
Ensure that file->external_ref points to TABLE object.
storage/myisam/mi_extra.c:
Dummy test-if-killed for standalone
storage/myisam/mi_key.c:
If ma_check_index_cond() fails, set my_errno and info->lastpos
storage/myisam/mi_rkey.c:
Ensure that info->lastpos= HA_OFFSET_ERROR in case of error
Release/re-acquire locks to give queries that wait on them a chance to make progress
Check if the query was killed, and return immediately if it was
Reorder code to do less things in case of error.
Added missing fast_mi_writeinfo()
storage/myisam/mi_rnext.c:
Check if the query was killed, and return immediately if it was
Simplify old ICP code
Added missing fast_ma_writeinfo(info)
storage/myisam/mi_rnext_same.c:
Check if the query was killed, and return immediately if it was
Added missing fast_mi_writeinfo(info)
storage/myisam/mi_rprev.c:
Check if the query was killed, and return immediately if it was
Simplify error handling of ICP
Added missing fast_mi_writeinfo(info)
storage/myisam/mi_search.c:
Give error message if we find a wrong key
storage/myisam/mi_static.c:
Added pointer to test-if-killed function
storage/myisam/myisamdef.h:
New prototypes
storage/xtradb/handler/ha_innodb.cc:
Added DB_SEARCH_ABORTED_BY_USER and ha_innobase::is_thd_killed()
Check if the query was killed, and return immediately if it was
storage/xtradb/handler/ha_innodb.h:
Added prototype
storage/xtradb/include/db0err.h:
Added DB_SEARCH_ABORTED_BY_USER
storage/xtradb/include/row0mysql.h:
Added possible ICP errors
storage/xtradb/row/row0sel.c:
Use ICP errors instead of constants.
Detect if killed and return B_SEARCH_ABORTED_BY_USER
The patch also adjusts several instable test results
to order the result.
Analysis:
The function prev_record_reads() may skip (jump over)
some query plan nodes where record_count < 1. At the
same time, even though get_partial_join_cost() uses
all first N plan nodes after the last constant table,
it may produce a smaller record_count than
prev_record_reads(), because the record count for
some plan nodes may be < 1, and these nodes may not
participate in prev_record_reads.
Solution:
The current solution is to treat the result of
get_partial_join_cost() as the upper bound for the
total number of unique lookup keys.
This bug extends the fix for LP BUG#715027 to cover one
more case of an Item being transformed, and its property
Item::with_subselect not being updated because
quick_fix_fields doesn't recalculate any properties.
Analysis:
Before calling:
write_record= (select->skip_record(thd) > 0);
the function find_all_keys needs to restore the original read/write
sets of the table that is sorted if the condition select->cond
contains a subquery.
This didn't happen in this test case because the flag "with_subselect"
was not set properly for select->cond.
The reason for the flag not being set properly, was that this condition
was rewritten by add_cond_and_fix() inside make_join_select() by:
/* Add conditions added by add_not_null_conds(). */
if (tab->select_cond)
add_cond_and_fix(thd, &tmp, tab->select_cond);
However, the function add_cond_and_fix() called the shortcut method
Item::quick_fix_field() that didn't update the "with_subselect"
property.
Solution:
Call the complete Item::fix_fields() to update all Item properties,
including "with_subselect".
Analysis:
The failed assert is a result of calling Item_sum_distinct::clear()
on an incomplete object for which Item_sum_distinct::setup() was
not yet called.
The reason is that JOIN::exec for the outer query calls JOIN::reinit()
for all its subqueries, which in turn calls clear() for all aggregate
functions of the subqueries. The call stack is:
mysql_explain_union -> mysql_select -> JOIN::exec -> select_desribe ->
mysql_explain_union -> mysql_select -> JOIN::reinit
This assert doesn't fail in the main 5.3 because constant subqueries
are being executed during the optimize phase of the outer query,
thus the Unique object is created before calling JOIN::exec for the
outer query, and Item_sum_distinct::clear() actually cleans the
Unique object.
Solution:
The best solution is the obvious one - substitute the assert with
a test whether Item_sum_distinct::tree is NULL.
Analysis:
The crash in EXPLAIN resulted from an attempt to print the
name of the internal temporary table created to compute
distinct for the innermost subquery of the test case.
Such tables do not have a corresponding TABLE_LIST (table
reference), hence the crash. The reason for this was that
the subquery was executed as part of constant condition
evaluation before EXPLAIN attempts to print the table name.
During the subquery execution, the subquery JOIN_TAB and
its table are substituted by a temporary table in
make_simple_join.
Solution:
Similar to the analogous case for other Items than the
IS NULL function, do not evaluate expensive constant
conditions.
