Analysis:
Range analysis discoveres that the query can be executed via loose index scan for GROUP BY.
Later, GROUP BY analysis fails to confirm that the GROUP operation can be computed via an
index because there is no logic to handle duplicate field references in the GROUP clause.
As a result the optimizer produces an inconsistent plan. It constructs a temporary table,
but on the other hand the group fields are not set to point there.
Solution:
Make loose scan analysis work in sync with order by analysis. In the case of duplicate
columns loose scan will not be applicable. This limitation will be lifted in 10.0 by
removing duplicate columns.
If the setting of system variables does not allow to use join buffer
for a join query with GROUP BY <f1,...> / ORDER BY <f1,...> then
filesort is not needed if the first joined table is scanned in
the order compatible with order specified by the list <f1,...>.
fix: don't call field->val_decimal() if the field->is_null()
because the buffer at field->ptr might not hold a valid decimal value
sql/item_sum.cc:
do not call field->val_decimal() if the field->is_null()
storage/maria/ma_blockrec.c:
cleanup
storage/maria/ma_rrnd.c:
cleanup
strings/decimal.c:
typo
Optimization of aggregate functions detected constant under max() and evalueted it, but condition in the WHWRE clause (which is always FALSE) was not taken into account
The patch backports two patches from mysql 5.6:
- BUG#12640437: USING SQL_BUFFER_RESULT RESULTS IN A DIFFERENT QUERY OUTPUT
- Bug#12578908: SELECT SQL_BUFFER_RESULT OUTPUTS TOO MANY ROWS WHEN GROUP IS OPTIMIZED AWAY
Original comment:
-----------------
3714 Jorgen Loland 2012-03-01
BUG#12640437 - USING SQL_BUFFER_RESULT RESULTS IN A DIFFERENT
QUERY OUTPUT
For all but simple grouped queries, temporary tables are used to
resolve grouping. In these cases, the list of grouping fields is
stored in the temporary table and grouping is resolved
there (e.g. by adding a unique constraint on the involved
fields). Because of this, grouping is already done when the rows
are read from the temporary table.
In the case where a group clause may be optimized away, grouping
does not have to be resolved using a temporary table. However, if
a temporary table is explicitly requested (e.g. because the
SQL_BUFFER_RESULT hint is used, or the statement is
INSERT...SELECT), a temporary table is used anyway. In this case,
the temporary table is created with an empty group list (because
the group clause was optimized away) and it will therefore not
create groups. Since the temporary table does not take care of
grouping, JOIN::group shall not be set to false in
make_simple_join(). This was fixed in bug 12578908.
However, there is an exception where make_simple_join() should
set JOIN::group to false even if the query uses a temporary table
that was explicitly requested but is not strictly needed. That
exception is if the loose index scan access method (explain
says "Using index for group-by") is used to read into the
temporary table. With loose index scan, grouping is resolved
by the access method. This is exactly what happens in this bug.
The problem was in the code (update_const_equal_items()) which marked
index parts constant independently of the place where the equality was used.
In the test suite it marked t2_1.c part constant despite the fact that
it connected by OR with other expression.
Solution is to mark constant only top equalities connected with AND.
in EXPLAIN as select_type==MATERIALIZED.
Before, we had select_type==SUBQUERY and it was difficult to tell materialized
subqueries from uncorrelated scalar-context subqueries.
GROUPING BY FUNCTIONS.... (PART
The bug was introduced in a patch for bug 49897.
Problem: The assertion inserted by the original patch to guard against
zero-lenght sort keys during merge phase triggers also when the whole
set fits in memory.
Fix: Move assert so that it does not trigger if the whole set is in
memory.
mysql-test/r/group_by.result:
Add test for bug#11765254
mysql-test/t/group_by.test:
Add test for bug#11765254
sql/filesort.cc:
Move assertion
- Set the default
- Adjust the testcases so that 'new' tests are run with optimizations turned on.
- Pull out relevant tests from "irrelevant" tests and run them with optimizations on.
- Run range.test and innodb.test with both mrr=on and mrr=off
semijoin=on,firstmatch=on,loosescan=on
to
semijoin=off,firstmatch=off,loosescan=off
Adjust the testcases:
- Modify subselect*.test and join_cache.test so that all tests
use the same execution paths as before (i.e. optimizations that
are being tested are enabled)
- Let all other test files run with the new default settings (i.e.
with new optimizations disabled)
- Copy subquery testcases from these files into t/subselect_extra.test
which will run them with new optimizations enabled.
The loop that was looping over subqueries' references to outer field used a
local boolean variable to tell whether the field was grouped or not. But the
implementor failed to reset the variable after each iteration. Thus a field
that was not directly aggregated appeared to be.
Fixed by resetting the variable upon each new iteration.
- Fixed problem with oqgraph and 'make dist'
Note that after this merge we have a problem show in join_outer where we examine too many rows in one specific case (related to BUG#57024).
This will be fixed when mwl#128 is merged into 5.3.
Open issues:
- A better fix for #57688; Igor is working on this
- Test failure in index_merge_innodb.test ; Igor promised to look at this
- Some Innodb tests fails (need to merge with latest xtradb) ; Kristian promised to look at this.
- Failing tests: innodb_plugin.innodb_bug56143 innodb_plugin.innodb_bug56632 innodb_plugin.innodb_bug56680 innodb_plugin.innodb_bug57255
- Werror is disabled; Should be enabled after merge with xtradb.
plans or wrong results due to the fact that JOIN_CACHE functions
ignored the possibility of interleaving materialized semijoin
tables with tables whose records were stored in join buffers.
This fixes would become mostly unnecessary if the new code of
mwl 90 was merged into 5.3 right now.
Yet the fix the code of optimize_wo_join_buffering was needed
in any case.
Lines below which were added in the patch for Bug#56814 cause this crash:
+ if (table->table)
+ table->table->maybe_null= FALSE;
Consider following test case:
--
CREATE TABLE t1(f1 INT NOT NULL);
INSERT INTO t1 VALUES (16777214),(0);
SELECT COUNT(*) FROM t1 LEFT JOIN t1 t2
ON 1 WHERE t2.f1 > 1 GROUP BY t2.f1;
DROP TABLE t1;
--
We set TABLE::maybe_null to FALSE for t2 table
and in create_tmp_field() we create appropriate tmp table field
using create_tmp_field_from_item() function instead of
create_tmp_field_from_field. As a result we have
LONGLONG field. As we have GROUP BY clause we calculate
group buffer length, see calc_group_buffer().
Item from group list which is used for calculation
refer to the field from real tables and have LONG type.
So group buffer length become insufficient for storing of
LONGLONG value. It leads to overwriting of wrong memory
area in do_field_int() function which is called from
end_update().
After some investigation I found out that
create_tmp_field_from_item() is used only for OLAP
grouping and can not be used for common grouping
as it could be an incompatibility between tmp
table fields and group buffer length.
We can not remove create_tmp_field_from_item() call from
create_tmp_field as OLAP needs it and we can not use this
function for common grouping. So we should remove setting
TABLE::maybe_null to FALSE from simplify_joins().
In this case we'll get wrong behaviour of
list_contains_unique_index() back. To fix it we
could use Field::real_maybe_null() check instead of
Field::maybe_null() and add addition check of
TABLE_LIST::outer_join.
mysql-test/r/group_by.result:
test case
mysql-test/r/join_outer.result:
test case
mysql-test/t/group_by.test:
test case
mysql-test/t/join_outer.test:
test case
sql/sql_select.cc:
--remove wrong code
--use Field::real_maybe_null() check instead of
Field::maybe_null() and add addition check of
TABLE_LIST::outer_join
Applied the fix for bug #47217 from the mysql-6.0 codebase.
The patch adds not null predicates generated for the left parts
of the equality predicates used for ref accesses. This is done
for such predicates both in where conditions and on conditions.
For the where conditions the not null predicates were generated
but in 5.0/5.1 they actually never were used due to some lame
merge from 4.1 to 5.0. The fix for bug #47217 made these
predicates to be used in the condition pushed to the tables.
Yet only this patch generates not null predicates for equality
predicated from on conditions of outer joins.
This patch introduces a performance regression that can be
observed on a test case from null_key.test. The regression
will disappear after the fix for bug #57024 from mariadb-5.1
is pulled into mariadb-5.3.
The patch contains many changes in the outputs of the EXPLAIN
commands since generated not null predicates are considered as
parts of the conditions pushed to join tables and may add
'Usingwhere' in some rows of EXPLAINs where there used
to be no such comments.
- Changed to still use bcmp() in certain cases becasue
- Faster for short unaligneed strings than memcmp()
- Bettern when using valgrind
- Changed to use my_sprintf() instead of sprintf() to get higher portability for old systems
- Changed code to use MariaDB version of select->skip_record()
- Removed -%::SCCS/s.% from Makefile.am:s to remove automake warnings
mysql-test/r/group_by.result:
Added test that showed problems that no_rows_in_results() didn't work for expressions
mysql-test/r/subselect4.result:
Test case for LP#612894
mysql-test/t/group_by.test:
Added test that showed problems that no_rows_in_results() didn't work for expressions
mysql-test/t/subselect4.test:
Test case for LP#612894
sql/item.h:
Added restore_to_before_no_rows_in_result()
Added function processor for no_rows_in_results() and restore_to_before_no_rows_in_results() to ensure it works with functions
Fix that above functions are handled by Item_ref()
sql/item_func.h:
Ensure that no_rows_in_results() and restore_to_before_no_rows_in_result() are called for all function arguments
sql/item_sum.cc:
Added restore_to_before_no_rows_in_result() to restore settings after Item_sum_hybrid::no_rows_in_result() was called.
This is needed to handle the case where we have made 'make_const()' on the item in opt_sum(), but the item will be reused again in a sub query.
Ignore multiple calls to no_rows_in_result() as Item_ref is calling it twice.
sql/item_sum.h:
Added restore_to_before_no_rows_in_result();
sql/sql_select.cc:
Added reset of no_rows_in_result() for JOIN::reinit()
sql/sql_select.h:
Added marker if no_rows_in_result() is called.
In order to be able to check if the set of the grouping fields in a
GROUP BY has changed (and thus to start a new group) the optimizer
caches the current values of these fields in a set of Cached_item
derived objects.
The Cached_item_str, used for caching varchar and TEXT columns,
is limited in length by the max_sort_length variable.
A String buffer to store the value with an alloced length of either
the max length of the string or the value of max_sort_length
(whichever is smaller) in Cached_item_str's constructor.
Then, at compare time the value of the string to compare to was
truncated to the alloced length of the string buffer inside
Cached_item_str.
This is all fine and valid, but only if you're not assigning
values near or equal to the alloced length of this buffer.
Because when assigning values like this the alloced length is
rounded up and as a result the next set of data will not match the
group buffer, thus leading to wrong results because of the changed
alloced_length.
Fixed by preserving the original maximum length in the
Cached_item_str's constructor and using this instead of the
alloced_length to limit the string to compare to.
Test case added.
NULL from outer join query
Problem: optimising MIN/MAX() queries without GROUP BY clause
by replacing the aggregate expression with a constant, we may set it
to NULL disregarding the fact that there may be outer joins involved.
Fix: don't replace MIN/MAX() with NULL if there're outer joins.
Note: the fix itself is just
- if (!count)
+ if (!count && !outer_tables)
set to NULL
The rest of the patch eliminates repeated code to improve speed
and for easy maintenance of the code.
mysql-test/r/group_by.result:
Fix for bug#52051: Aggregate functions incorrectly returns
NULL from outer join query
- test result.
mysql-test/t/group_by.test:
Fix for bug#52051: Aggregate functions incorrectly returns
NULL from outer join query
- test case.
sql/opt_sum.cc:
Fix for bug#52051: Aggregate functions incorrectly returns
NULL from outer join query
- optimising MIN/MAX() queries without GROUP BY clause by
replacing them with a constant, take into account that
there're may be outer joins involved.
- repeated code for MIN/MAX optimization in the opt_sum_query()
eliminated by introducing new functions that read MIN/MAX values
using index and combining MIN/MAX cases to one.
The problem is that not all column names retrieved from a SELECT
statement can be used as view column names due to length and format
restrictions. The server failed to properly check the conformity
of those automatically generated column names before storing the
final view definition on disk.
Since columns retrieved from a SELECT statement can be anything
ranging from functions to constants values of any format and length,
the solution is to rewrite to a pre-defined format any names that
are not acceptable as a view column name.
The name is rewritten to "Name_exp_%u" where %u translates to the
position of the column. To avoid this conversion scheme, define
explict names for the view columns via the column_list clause.
Also, aliases are now only generated for top level statements.
mysql-test/include/view_alias.inc:
Add test case for Bug#40277
mysql-test/r/compare.result:
Bug#40277: SHOW CREATE VIEW returns invalid SQL
mysql-test/r/group_by.result:
Bug#40277: SHOW CREATE VIEW returns invalid SQL
mysql-test/r/ps.result:
Bug#40277: SHOW CREATE VIEW returns invalid SQL
mysql-test/r/subselect.result:
Bug#40277: SHOW CREATE VIEW returns invalid SQL
mysql-test/r/subselect3.result:
Bug#40277: SHOW CREATE VIEW returns invalid SQL
mysql-test/r/type_datetime.result:
Bug#40277: SHOW CREATE VIEW returns invalid SQL
mysql-test/r/union.result:
Bug#40277: SHOW CREATE VIEW returns invalid SQL
mysql-test/r/view.result:
Add test case result for Bug#40277
mysql-test/r/view_alias.result:
Add test case result for Bug#40277
mysql-test/t/view_alias.test:
Add test case for Bug#40277
sql/sql_view.cc:
Check if auto generated column names are conforming. Also, the
make_unique_view_field_name function is not used as it uses the
original name to construct a new one, which does not work if the
name is invalid.
