We assume all around the code that null_value==true is in sync
with NULL value returned by val_str()/val_decimal().
Item_sum_sum::val_decimal() erroneously returned a non-NULL value together
with null_value set to true. Fixing to return NULL instead.
Correct fix for this bug.
The problem was that Item_func_group_concat() was calling
setup_order(), passing args as the second argument,
ref_pointer_array. While ref_pointer_array should have free
space at the end, as setup_order() can append elements to it.
In this particular case args[] elements were overwritten when
setup_order() was pushing new elements into ref_pointer_array.
execution of PS
GROUP_CONCAT() with ORDER BY column position may crash server on PS reexecution.
The problem was that arguments array of GROUP_CONCAT() was adjusted to point to
temporary elements (resolved ORDER BY fields) during first execution.
This patch expands rev. 08763096cb to restore original arguments array as well.
3RD EXECUTION OF PS
Problem:
When order by is by a column number for a group concat function
which has an outer reference, server fails in case of prepared
statements on the third execution
Analysis:
When a group concat function has order by, the fields in order by
are not resolved until execution if the input is a column number.
During execution they get resolved after the temp table gets created.
As a result they will be pointing to temp table fields which are
runtime created objects. This results in dangling pointers leading
to server failure.
Solution:
Reset the pointers for the order by fields to point to the original
arguments after execution as they are invalid.
Done in Item_func_group_concat::cleanup.
Problem was that we used cache_table in some cases where it was not initialized
mysql-test/r/func_group.result:
Added test case
mysql-test/t/func_group.test:
Added test case
sql/item.cc:
Don't use cached_table if not set
sql/item_sum.cc:
Don't use cached_table
Bug#17867117 - ERROR RESULT WHEN "COUNT + DISTINCT + CASE WHEN" NEED MERGE_WALK
Problem:
COUNT DISTINCT gives incorrect result when it uses a Unique
Tree and its last inserted record has null value.
Here is how COUNT DISTINCT is processed, given that this query is not
using loose index scan.
When a row is produced as a result of joining tables (there is only
one table here), we store the SELECTed value in a Unique tree. This
allows elimination of any duplicates, and thus implements DISTINCT.
When we have processed all rows like this, we walk the Unique tree,
counting its elements, in Aggregator_distinct::endup() (tree->walk());
for each element we call Item_sum_count::add(). Such function wants to
ignore any NULL value, for that it checks item_sum -> args[0] ->
null_value. It is a mistake: when walking the Unique tree, the value
to be aggregated is not item_sum ->args[0] but rather table ->
field[0].
Solution:
instead of item_sum -> args[0] -> null_value, use arg_is_null(), which
knows where to look (like in fix for bug 57932).
As a consequence of this solution, we have to make arg_is_null() a
little more general:
1) Because it was so far only used for AVG() (which always has a
single argument), this function was looking at a single argument; now
that it has to work with COUNT(DISTINCT expression1,expression2), it
must look at all arguments.
2) Because we start using arg_is_null () for COUNT(DISTINCT), i.e. in
Item_sum_count::add (), it implies that we are also using it for
COUNT(no DISTINCT) (same add ()). For COUNT(no DISTINCT), the
nullness to check is that of item_sum -> args[0]. But the null_value
of such item is reliable only if val_*() has been called on it. So far
arg_is_null() was always used after a call to arg_val*(), so could
rely on null_value; but for COUNT, there is no call to arg_val*(), so
arg_is_null() has to call is_null() instead.
Testcase for 16539979 by Neeraj. Testcase for 17867117 contributed by
Xiaobin Lin from Taobao.
Bug#17867117 - ERROR RESULT WHEN "COUNT + DISTINCT + CASE WHEN" NEED MERGE_WALK
Problem:
COUNT DISTINCT gives incorrect result when it uses a Unique
Tree and its last inserted record has null value.
Here is how COUNT DISTINCT is processed, given that this query is not
using loose index scan.
When a row is produced as a result of joining tables (there is only
one table here), we store the SELECTed value in a Unique tree. This
allows elimination of any duplicates, and thus implements DISTINCT.
When we have processed all rows like this, we walk the Unique tree,
counting its elements, in Aggregator_distinct::endup() (tree->walk());
for each element we call Item_sum_count::add(). Such function wants to
ignore any NULL value, for that it checks item_sum -> args[0] ->
null_value. It is a mistake: when walking the Unique tree, the value
to be aggregated is not item_sum ->args[0] but rather table ->
field[0].
Solution:
instead of item_sum -> args[0] -> null_value, use arg_is_null(), which
knows where to look (like in fix for bug 57932).
As a consequence of this solution, we have to make arg_is_null() a
little more general:
1) Because it was so far only used for AVG() (which always has a
single argument), this function was looking at a single argument; now
that it has to work with COUNT(DISTINCT expression1,expression2), it
must look at all arguments.
2) Because we start using arg_is_null () for COUNT(DISTINCT), i.e. in
Item_sum_count::add (), it implies that we are also using it for
COUNT(no DISTINCT) (same add ()). For COUNT(no DISTINCT), the
nullness to check is that of item_sum -> args[0]. But the null_value
of such item is reliable only if val_*() has been called on it. So far
arg_is_null() was always used after a call to arg_val*(), so could
rely on null_value; but for COUNT, there is no call to arg_val*(), so
arg_is_null() has to call is_null() instead.
Testcase for 16539979 by Neeraj. Testcase for 17867117 contributed by
Xiaobin Lin from Taobao.
CONTAINING NULL
Problem:-
In MySQL, We can obtain the number of distinct expression
combinations that do not contain NULL by giving a list of
expressions in COUNT(DISTINCT).
However rows with NULL values are
incorrectly included in the count when loose index scan is
used.
Analysis:-
In case of loose index scan, we check whether the field is null or
not and increase the count in Item_sum_count::add().
But there we are checking for the first field in COUNT(DISTINCT),
not for every field. This is causing an incorrect result.
Solution:-
Check all field in Item_sum_count::add(), whether there values
are null or not. Then only increment the count.
******
Bug#17222452 - SELECT COUNT(DISTINCT A,B) INCORRECTLY COUNTS ROWS
CONTAINING NULL
Problem:-
In MySQL, We can obtain the number of distinct expression
combinations that do not contain NULL by giving a list of
expressions in COUNT(DISTINCT).
However rows with NULL values are
incorrectly included in the count when loose index scan is
used.
Analysis:-
In case of loose index scan, we check whether the field is null or
not and increase the count in Item_sum_count::add().
But there we are checking for the first field in COUNT(DISTINCT),
not for every field. This is causing an incorrect result.
Solution:-
Check all field in Item_sum_count::add(), whether there values
are null or not. Then only increment the count.
CONTAINING NULL
Problem:-
In MySQL, We can obtain the number of distinct expression
combinations that do not contain NULL by giving a list of
expressions in COUNT(DISTINCT).
However rows with NULL values are
incorrectly included in the count when loose index scan is
used.
Analysis:-
In case of loose index scan, we check whether the field is null or
not and increase the count in Item_sum_count::add().
But there we are checking for the first field in COUNT(DISTINCT),
not for every field. This is causing an incorrect result.
Solution:-
Check all field in Item_sum_count::add(), whether there values
are null or not. Then only increment the count.
******
Bug#17222452 - SELECT COUNT(DISTINCT A,B) INCORRECTLY COUNTS ROWS
CONTAINING NULL
Problem:-
In MySQL, We can obtain the number of distinct expression
combinations that do not contain NULL by giving a list of
expressions in COUNT(DISTINCT).
However rows with NULL values are
incorrectly included in the count when loose index scan is
used.
Analysis:-
In case of loose index scan, we check whether the field is null or
not and increase the count in Item_sum_count::add().
But there we are checking for the first field in COUNT(DISTINCT),
not for every field. This is causing an incorrect result.
Solution:-
Check all field in Item_sum_count::add(), whether there values
are null or not. Then only increment the count.
!TABLES->NEXT_NAME_RESOLUTION_TABLE) || !TAB
Problem:
The context info of select query gets corrupted when a query
with group_concat having order by is present in an order by
clause of the select query. As a result, server crashes with
an assert.
Analysis:
While parsing order by for group_concat, it is presumed that
it is always present before the actual order by for the
select query.
As a result, parser uses select->order_list to populate the
order by items of group_concat and creates a select->gorder_list
to which select->order_list is copied onto. Once this is done,
it empties the select->order_list.
In the case presented in the bugpage, as order by is already
parsed when group_concat's order by is encountered, parser
presumes that it is the second order by in the select query
and creates fake_lex_unit which results in the change of
context info.
Solution:
Make group_concat's order by parsing independent of the select
sql/item_sum.cc:
Change the argument as, select->gorder_list is not pointer anymore
sql/item_sum.h:
Change the argument as, select->gorder_list is not pointer anymore
sql/mysql_priv.h:
Parsing for group_concat's order by is made independent.
As a result, add_order_to_list cannot be used anymore.
sql/sql_lex.cc:
Parsing for group_concat's order by is made independent.
As a result, add_order_to_list cannot be used anymore.
sql/sql_lex.h:
Parsing for group_concat's order by is made independent.
As a result, add_order_to_list cannot be used anymore.
sql/sql_yacc.yy:
Make group_concat's order by parsing independent of the select
queries order by.
!TABLES->NEXT_NAME_RESOLUTION_TABLE) || !TAB
Problem:
The context info of select query gets corrupted when a query
with group_concat having order by is present in an order by
clause of the select query. As a result, server crashes with
an assert.
Analysis:
While parsing order by for group_concat, it is presumed that
it is always present before the actual order by for the
select query.
As a result, parser uses select->order_list to populate the
order by items of group_concat and creates a select->gorder_list
to which select->order_list is copied onto. Once this is done,
it empties the select->order_list.
In the case presented in the bugpage, as order by is already
parsed when group_concat's order by is encountered, parser
presumes that it is the second order by in the select query
and creates fake_lex_unit which results in the change of
context info.
Solution:
Make group_concat's order by parsing independent of the select
Problem:
A select query inside a group_concat function having an
outer reference results in a crash.
Analysis:
In function Item_group_concat::add, we do not check if
return value of get_tmp_table_field can be NULL for
a non-const item. This can happen for a query with a
outer reference.
While resolving the outer reference in the query present
inside group_concat function, we set the "const_item_cache"
to false. As a result in the call to const_item() from
Item_func_group_concat::add, it returns false and goes on
to check if this can be NULL resulting in the crash.
get_tmp_table_field does not return NULL for Items of type
Item_field, Item_result_field and Item_ref.
For all other items, it returns NULL.
Solution:
Check for the return value of get_tmp_table_field before we
access field contents.
sql/item_sum.cc:
Check for the return value of get_tmp_table_field before accessing
Problem:
A select query inside a group_concat function having an
outer reference results in a crash.
Analysis:
In function Item_group_concat::add, we do not check if
return value of get_tmp_table_field can be NULL for
a non-const item. This can happen for a query with a
outer reference.
While resolving the outer reference in the query present
inside group_concat function, we set the "const_item_cache"
to false. As a result in the call to const_item() from
Item_func_group_concat::add, it returns false and goes on
to check if this can be NULL resulting in the crash.
get_tmp_table_field does not return NULL for Items of type
Item_field, Item_result_field and Item_ref.
For all other items, it returns NULL.
Solution:
Check for the return value of get_tmp_table_field before we
access field contents.
This bug was introduced by the patch for WL#3220.
If the memory allocated for the tree to store unique elements
to be counted is not big enough to include all of them then
an external file is used to store the elements.
The unique elements are guaranteed not to be nulls. So, when
reading them from the file we don't have to care about the null
flags of the read values. However, we should remove the flag
at the very beginning of the process. If we don't do it and
if the last value written into the record buffer for the field
whose distinct values needs to be counted happens to be null,
then all values read from the file are considered to be nulls
and are not counted in.
The fix does not remove a possible null flag for the read values.
Rather it just counts the values in the same way it was done
before WL #3220.
Item_func_group_concat::copy_or_same() creates a copy of original object.
It also creates a copy of ORDER structure because ORDER struct elements may
be modified in find_order_in_list() called from Item_func_group_concat::setup().
As ORDER copy is created using memcpy, ORDER::next elements point to original
ORDER structs. Thus find_order_in_list() called from EXECUTE stmt modifies
ordinal ORDER item pointers so they point to runtime items, these items are
freed after execution, so original ORDER structure becomes invalid.
The fix is to properly update ORDER::next fields so that they point to
new ORDER elements.
sql/item_sum.cc:
update ORDER::next fields so that they point to new ORDER elements.
Item_func_group_concat::copy_or_same() creates a copy of original object.
It also creates a copy of ORDER structure because ORDER struct elements may
be modified in find_order_in_list() called from Item_func_group_concat::setup().
As ORDER copy is created using memcpy, ORDER::next elements point to original
ORDER structs. Thus find_order_in_list() called from EXECUTE stmt modifies
ordinal ORDER item pointers so they point to runtime items, these items are
freed after execution, so original ORDER structure becomes invalid.
The fix is to properly update ORDER::next fields so that they point to
new ORDER elements.
This bug could result in returning 0 for the expressions of the form
<aggregate_function>(distinct field) when the system variable
max_heap_table_size was set to a small enough number.
It happened because the method Unique::walk() did not support
the case when more than one pass was needed to merge the trees
of distinct values saved in an external file.
Backported a fix in grant_lowercase.test from mariadb 5.5.
Analysis:
---------
When the server is out of memory, an error is raised
to indicate the same. Handling the error requires
more memory to be allocated which fails, hence the
error handling loops in a recursion and causes the
server to crash.
Fix:
---
a) Prevents pushing the 'out of memory' error condition
to the diagnostic area as it requires memory allocation.
GET DIAGNOSTICS, SHOW WARNINGS and SHOW ERRORS statements
will not show information about this error. However the
'out of memory' error is returned to the client.
b) It sets the ME_FATALERROR flag when 'out of memory' errors
are reported (for places where the flag is not already set).
This flag prevents activation of SP error handlers which also
require memory allocation and therefore are likely to fail.
Analysis:
---------
When the server is out of memory, an error is raised
to indicate the same. Handling the error requires
more memory to be allocated which fails, hence the
error handling loops in a recursion and causes the
server to crash.
Fix:
---
a) Prevents pushing the 'out of memory' error condition
to the diagnostic area as it requires memory allocation.
GET DIAGNOSTICS, SHOW WARNINGS and SHOW ERRORS statements
will not show information about this error. However the
'out of memory' error is returned to the client.
b) It sets the ME_FATALERROR flag when 'out of memory' errors
are reported (for places where the flag is not already set).
This flag prevents activation of SP error handlers which also
require memory allocation and therefore are likely to fail.
Fixed MDEV-4002: Server crash or valgrind errors in Item_func_group_concat::setup and Item_func_group_concat::add
mysql-test/r/group_by.result:
Added test case for failing GROUP_CONCAT ... ROLLUP queries
mysql-test/t/group_by.test:
Added test case for failing GROUP_CONCAT ... ROLLUP queries
sql/item_sum.cc:
Fixed issue where field->table pointed to different temporary table than expected.
Ensure that order->next points to the right object (could cause problems with setup_order())
