execution of PS
GROUP_CONCAT() with ORDER BY column position may crash server on PS reexecution.
The problem was that arguments array of GROUP_CONCAT() was adjusted to point to
temporary elements (resolved ORDER BY fields) during first execution.
This patch expands rev. 08763096cb to restore original arguments array as well.
It is possible for Item_field to have a NULL field_name. This is true if
the Item_field is created based on a field in a temporary table that has
no name. It is thus necessary to do a null check before attempting a
strcmp.
sql/sql_insert.cc:
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
******
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
sql/sql_table.cc:
small cleanup
******
small cleanup
The bitmap of used tables must be evaluated for the select list of every
materialized derived table / view and saved in a dedicated field.
This is also applied to materialized subqueries.
The value of THD::used tables should be re-evaluated after merges
of views and derived tables into the main query.
Now it's done in the function SELECT_LEX::update_used_tables.
The re-evaluation of the 'used_table' bitmaps for the items
in HAVING, GROUP BY and ORDER BY clauses has been added as well.
Resolved all conflicts, bad merges and fixed a few minor bugs in the code.
Commented out the queries from multi_update, view, subselect_sj, func_str,
derived_view, view_grant that failed either with crashes in ps-protocol or
with wrong results.
The failures are clear indications of some bugs in the code and these bugs
are to be fixed.
This is a backport of
Bug #46860 Crash/segfault using EXPLAIN EXTENDED on query using UNION in subquery.
mysql-test/r/explain.result:
Remove the extended Note, since explain returns error.
New test case.
mysql-test/r/func_gconcat.result:
Remove the extended Note, since explain returns error.
mysql-test/r/subselect3.result:
Remove the extended Note, since explain returns error.
mysql-test/t/explain.test:
New test case.
sql/sql_parse.cc:
Skip the extended description if we have an error.
Explain fails at fix_fields stage and some items are left unfixed,
particulary Item_group_concat. Item_group_concat::orig_args field
is uninitialized in this case and Item_group_concat::print call
leads to crash.
The fix:
move the initialization of Item_group_concat::orig_args
into constructor.
mysql-test/r/func_gconcat.result:
test case
mysql-test/t/func_gconcat.test:
test case
sql/item_sum.cc:
move the initialization of Item_group_concat::orig_args
into constructor.
The problem is caused by bug49487 fix and became visible
after after bug56679 fix.
Items are cleaned up and set to unfixed state after filling derived table.
So we can not rely on item::fixed state in Item_func_group_concat::print
and we can not use 'args' array as items there may be cleaned up.
The fix is always to use orig_args array of items as it
always should contain the correct data.
mysql-test/r/func_gconcat.result:
test case
mysql-test/t/func_gconcat.test:
test case
sql/item_sum.cc:
The fix is always to use orig_args array of items.
prepared statements
Using GROUP_CONCAT() together with the WITH ROLLUP modifier
could crash the server.
The reason was a combination of several facts:
1. The Item_func_group_concat class stores pointers to ORDER
objects representing the columns in the ORDER BY clause of
GROUP_CONCAT().
2. find_order_in_list() called from
Item_func_group_concat::setup() modifies the ORDER objects so
that their 'item' member points to the arguments list
allocated in the Item_func_group_concat constructor.
3. In some cases (e.g. in JOIN::rollup_make_fields) a copy of
the original Item_func_group_concat object could be created by
using the Item_func_group_concat::Item_func_group_concat(THD
*thd, Item_func_group_concat *item) copy constructor. The
latter essentially creates a shallow copy of the source
object. Memory for the arguments array is allocated on
thd->mem_root, but the pointers for arguments and ORDER are
copied verbatim.
What happens in the test case is that when executing the query
for the first time, after a copy of the original
Item_func_group_concat object has been created by
JOIN::rollup_make_fields(), find_order_in_list() is called for
this new object. It then resolves ORDER BY by modifying the
ORDER objects so that they point to elements of the arguments
array which is local to the cloned object. When thd->mem_root
is freed upon completing the execution, pointers in the ORDER
objects become invalid. Those ORDER objects, however, are also
shared with the original Item_func_group_concat object which is
preserved between executions of a prepared statement. So the
first call to find_order_in_list() for the original object on
the second execution tries to dereference an invalid pointer.
The solution is to create copies of the ORDER objects when
copying Item_func_group_concat to not leave any stale pointers
in other instances with different lifecycles.
mysql-test/r/func_gconcat.result:
Test case for bug #54476.
mysql-test/t/func_gconcat.test:
Test case for bug #54476.
sql/item_sum.cc:
Copy the ORDER objects pointed to by the elements of the
'order' array in the copy constructor of
Item_func_group_concat.
sql/table.h:
Removed the unused 'item_copy' member of the ORDER class.
Conflicts:
Text conflict in mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result
Text conflict in sql/log.cc
Text conflict in sql/set_var.cc
Text conflict in sql/sql_class.cc
Problem: EXPLAIN EXTENDED was trying to resolve references to
freed temporary table fields for GROUP_CONCAT()'s ORDER BY arguments.
Fix: use stored original GROUP_CONCAT()'s arguments in such a case.
mysql-test/r/func_gconcat.result:
Fix for bug#52397: another crash with explain extended and group_concat
- test result.
mysql-test/t/func_gconcat.test:
Fix for bug#52397: another crash with explain extended and group_concat
- test case.
sql/item_sum.cc:
Fix for bug#52397: another crash with explain extended and group_concat
- use "pargs", printing ORDER BY arguments in the
Item_func_group_concat::print() instead of "order" to avoid
possible reference resolving to (freed) temporary table fields.
When EXPLAIN EXTENDED tries to print column names, it checks whether the
referenced table is CONST (in which case, the column's value rather than
its name will be printed). If no proper table is reference (i.e. because
a derived table was used that has since gone out of scope), this will fail
spectacularly.
This ports an equivalent of the fix for Bug 43354.
mysql-test/r/func_gconcat.result:
Show that EXPLAIN EXTENDED on a GROUP_CONCAT() on a derived table
no longer crashes the server.
mysql-test/t/func_gconcat.test:
Show that EXPLAIN EXTENDED on a GROUP_CONCAT() on a derived table
no longer crashes the server.
sql/item_sum.cc:
Do not de-ref what cannot be, that is, temp-tables that have gone away.
This is of questionable utility anyway, since our deref has the sole
purpose of checking whether the table is const (in which case, we'll
substitute the column with its value in EXPLAIN EXTENDED - that is all).
added:
include/ctype_numconv.inc
mysql-test/include/ctype_numconv.inc
mysql-test/r/ctype_binary.result
mysql-test/t/ctype_binary.test
Adding tests
modified:
mysql-test/r/bigint.result
mysql-test/r/case.result
mysql-test/r/create.result
mysql-test/r/ctype_cp1251.result
mysql-test/r/ctype_latin1.result
mysql-test/r/ctype_ucs.result
mysql-test/r/func_gconcat.result
mysql-test/r/func_str.result
mysql-test/r/metadata.result
mysql-test/r/ps_1general.result
mysql-test/r/ps_2myisam.result
mysql-test/r/ps_3innodb.result
mysql-test/r/ps_4heap.result
mysql-test/r/ps_5merge.result
mysql-test/r/show_check.result
mysql-test/r/type_datetime.result
mysql-test/r/type_ranges.result
mysql-test/r/union.result
mysql-test/suite/ndb/r/ps_7ndb.result
mysql-test/t/ctype_cp1251.test
mysql-test/t/ctype_latin1.test
mysql-test/t/ctype_ucs.test
mysql-test/t/func_str.test
Fixing tests
@ sql/field.cc
- Return str result using my_charset_numeric.
- Using real multi-byte aware str_to_XXX functions
to handle tricky charset values propely (e.g. UCS2)
@ sql/field.h
- Changing derivation of non-string field types to DERIVATION_NUMERIC.
- Changing binary() for numeric/datetime fields to always
return TRUE even if charset is not my_charset_bin. We need
this to keep ha_base_keytype() return HA_KEYTYPE_BINARY.
- Adding BINARY_FLAG into some fields, because it's not
being set automatically anymore with
"my_charset_bin to my_charset_numeric" change.
- Changing derivation for numeric/datetime datatypes to a weaker
value, to make "SELECT concat('string', field)" use character
set of the string literal for the result of the function.
@ sql/item.cc
- Implementing generic val_str_ascii().
- Using max_char_length() instead of direct read of max_length
to make "tricky" charsets like UCS2 work.
NOTE: in the future we'll possibly remove all direct reads of max_length
- Fixing Item_num::safe_charset_converter().
Previously it alligned binary string to
character string (for example by adding leading 0x00
when doing binary->UCS2 conversion). Now it just
converts from my_charset_numbner to "tocs".
- Using val_str_ascii() in Item::get_time() to make UCS2 arguments work.
- Other misc changes
@ sql/item.h
- Changing MY_COLL_CMP_CONV and MY_COLL_ALLOW_CONV to
bit operations instead of hard-coded bit masks.
- Addding new method DTCollation.set_numeric().
- Adding new methods to Item.
- Adding helper functions to make code look nicer:
agg_item_charsets_for_string_result()
agg_item_charsets_for_comparison()
- Changing charset for Item_num-derived items
from my_charset_bin to my_charset_numeric
(which is an alias for latin1).
@ sql/item_cmpfunc.cc
- Using new helper functions
- Other misc changes
@ sql/item_cmpfunc.h
- Fixing strcmp() to return max_length=2.
Previously it returned 1, which was wrong,
because it did not fit '-1'.
@ sql/item_func.cc
- Using new helper functions
- Other minor changes
@ sql/item_func.h
- Removing unused functions
- Adding helper functions
agg_arg_charsets_for_string_result()
agg_arg_charsets_for_comparison()
- Adding set_numeric() into constructors of numeric items.
- Using fix_length_and_charset() and fix_char_length()
instead of direct write to max_length.
@ sql/item_geofunc.cc
- Changing class for Item_func_geometry_type and
Item_func_as_wkt from Item_str_func to
Item_str_ascii_func, to make them return UCS2 result
properly (when character_set_connection=ucs2).
@ sql/item_geofunc.h
- Changing class for Item_func_geometry_type and
Item_func_as_wkt from Item_str_func to
Item_str_ascii_func, to make them return UCS2 result
properly (when @@character_set_connection=ucs2).
@ sql/item_strfunc.cc
- Implementing Item_str_func::val_str().
- Renaming val_str to val_str_ascii for some items,
to make them work with UCS2 properly.
- Using new helper functions
- All single-argument functions that expect string
result now call this method:
agg_arg_charsets_for_string_result(collation, args, 1);
This enables character set conversion to @@character_set_connection
in case of pure numeric input.
@ sql/item_strfunc.h
- Introducing Item_str_ascii_func - for functions
which return pure ASCII data, for performance purposes,
as well as for the cases when the old implementation
of val_str() was heavily 8-bit oriented and implementing
a UCS2-aware version is tricky.
@ sql/item_sum.cc
- Using new helper functions.
@ sql/item_timefunc.cc
- Using my_charset_numeric instead of my_charset_bin.
- Using fix_char_length(), fix_length_and_charset()
and fix_length_and_charset_datetime()
instead of direct write to max_length.
- Using tricky-charset aware function str_to_time_with_warn()
@ sql/item_timefunc.h
- Using new helper functions for charset and length initialization.
- Changing base class for Item_func_get_format() to make
it return UCS2 properly (when character_set_connection=ucs2).
@ sql/item_xmlfunc.cc
- Using new helper function
@ sql/my_decimal.cc
- Adding a new DECIMAL to CHAR converter
with real multibyte support (e.g. UCS2)
@ sql/mysql_priv.h
- Introducing a new derivation level for numeric/datetime data types.
- Adding macros for my_charset_numeric and MY_REPERTOIRE_NUMERIC.
- Adding prototypes for str_set_decimal()
- Adding prototypes for character-set aware str_to_xxx() functions.
@ sql/protocol.cc
- Changing charsetnr to "binary" client-side metadata for
numeric/datetime data types.
@ sql/time.cc
- Adding to_ascii() helper function, to convert a string
in any character set to ascii representation. In the
future can be extended to understand digits written
in various non-Latin word scripts.
- Adding real multy-byte character set aware versions for str_to_XXXX,
to make these these type of queries work correct:
INSERT INTO t1 SET datetime_column=ucs2_expression;
@ strings/ctype-ucs2.c
- endptr was not calculated correctly. INSERTing of UCS2
values into numeric columns returned warnings about
truncated wrong data.
into magare.gmz:/home/kgeorge/mysql/work/B35298-5.1-bugteam
mysql-test/r/func_gconcat.result:
Auto merged
mysql-test/t/func_gconcat.test:
Auto merged
sql/item_sum.cc:
Auto merged
configure.in:
merge 5.0-bugteam to 5.1-bugteam
The bug is a regression introduced by the patch for bug32798.
The code in Item_func_group_concat::clear() relied on the 'distinct'
variable to check if 'unique_filter' was initialized. That, however,
is not always valid because Item_func_group_concat::setup() can do
shortcuts in some cases w/o initializing 'unique_filter'.
Fixed by checking the value of 'unique_filter' instead of 'distinct'
before dereferencing.
mysql-test/r/func_gconcat.result:
Added test cases for bugs #35298 and #36024.
mysql-test/t/func_gconcat.test:
Added test cases for bugs #35298 and #36024.
sql/item_sum.cc:
Check if unique_filter != NULL before dereferencing it. Non-zero value
of distinct does not always mean that unique_filter is initialized
because Item_func_group_concat::setup() can do shortcuts is some cases
into magare.gmz:/home/kgeorge/mysql/work/B34747-5.1-opt
libmysql/libmysql.c:
Auto merged
libmysqld/lib_sql.cc:
Auto merged
mysql-test/r/func_gconcat.result:
Auto merged
mysql-test/t/func_gconcat.test:
Auto merged
sql/item_sum.cc:
Auto merged
sql/log.cc:
Auto merged
sql/sql_yacc.yy:
Auto merged
mysql-test/r/subselect.result:
merged bug 34747 and 33266 5.0-opt->5.1-opt
mysql-test/t/subselect.test:
merged bug 34747 and 33266 5.0-opt->5.1-opt
Was a double-free of the Unique member of Item_func_group_concat.
This was not causing a crash because the Unique is a descendent of
Sql_alloc.
Fixed to free the Unique only if it was allocated for the instance
of Item_func_group_concat it was referenced from
mysql-test/r/func_gconcat.result:
Bug #34747: test case
mysql-test/t/func_gconcat.test:
Bug #34747: test case
sql/item_sum.cc:
Bug #34747: free the Unique only if it was allocated
for this instance of Item_func_group_concat
into linux-st28.site:/home/martin/mysql/src/bug32798-united/my51-bug32798-united
mysql-test/r/func_gconcat.result:
Auto merged
mysql-test/t/func_gconcat.test:
Auto merged
sql/item_sum.cc:
Manual merge, Field::cmp has argument type const uchar* in 5.1
sql/item_sum.h:
manual merge
with null values
For queries containing GROUP_CONCAT(DISTINCT fields ORDER BY fields), there
was a limitation that the DISTINCT fields had to be the same as ORDER BY
fields, owing to the fact that one single sorted tree was used for keeping
track of tuples, ordering and uniqueness. Fixed by introducing a second
structure to handle uniqueness so that the original structure has only to
order the result.
mysql-test/r/func_gconcat.result:
Bug#32798:
- Wrong test result turned correct after fix.
- Correct test result
mysql-test/t/func_gconcat.test:
Bug#32798: Test case
sql/item_sum.cc:
Bug#32798: Implementation of fix. Dead code removal.
- removed comment describing this bug
- replaced body of function group_concat_key_cmp_with_distinct
- removed function group_concat_key_cmp_with_distinct_and_order
- Added a Unique object to maintain uniqueness of values.
sql/item_sum.h:
Bug#32798: Declarations and comments.
into mysql.com:/home/gluh/MySQL/Merge/5.1-opt
mysql-test/r/func_gconcat.result:
Auto merged
mysql-test/t/func_gconcat.test:
Auto merged
sql/item_sum.cc:
Auto merged
The fix is a copy of Martin Friebe's suggestion.
added testing for no_appended which will be false if anything,
including the empty string is in result
mysql-test/r/func_gconcat.result:
test result
mysql-test/t/func_gconcat.test:
test case
sql/item_sum.cc:
added testing for no_appended which will be False if anything,
including the empty string is in result
into mysql.com:/home/ram/work/b31154/b31154.5.1
mysql-test/r/func_gconcat.result:
Auto merged
mysql-test/t/func_gconcat.test:
Auto merged
sql/item_sum.cc:
Auto merged
Problem: GROUP_CONCAT(DISTINCT BIT_FIELD...) uses a tree to store keys;
which are constructed using a temporary table fields,
see Item_func_group_concat::setup().
As a) we don't store null bits in the tree where the bit fields store parts
of their data and b) there's no method to properly compare two table records
we've got problem.
Fix: convert BIT fields to INT in the temporary table used.
mysql-test/r/func_gconcat.result:
Fix for bug #31154: field.h:1649: virtual int Field_bit::cmp(const uchar*, const uchar*): Assertion
- test result.
mysql-test/t/func_gconcat.test:
Fix for bug #31154: field.h:1649: virtual int Field_bit::cmp(const uchar*, const uchar*): Assertion
- test case.
sql/item_sum.cc:
Fix for bug #31154: field.h:1649: virtual int Field_bit::cmp(const uchar*, const uchar*): Assertion
- force the create_tmp_table() to convert BIT columns to INT
in order to be able to compare records containing BIT fields.
into gleb.loc:/home/uchum/work/bk/5.1-opt
mysql-test/r/func_gconcat.result:
Auto merged
mysql-test/t/func_gconcat.test:
Auto merged
sql/item_sum.cc:
Auto merged
a temporary table.
The result string of the Item_func_group_concat wasn't initialized in the
copying constructor of the Item_func_group_concat class. This led to a
wrong charset of GROUP_CONCAT result when the select employs a temporary
table.
The copying constructor of the Item_func_group_concat class now correctly
initializes the charset of the result string.
mysql-test/t/func_gconcat.test:
Added a test case for the bug#29850: Wrong charset of the GROUP_CONCAT result
when the select employs a temporary table.
mysql-test/r/func_gconcat.result:
Added a test case for the bug#29850: Wrong charset of the GROUP_CONCAT result
when the select employs a temporary table.
sql/item_sum.cc:
Bug#29850: Wrong charset of GROUP_CONCAT result when the select employs
a temporary table.
The copying constructor of the Item_func_group_concat class now correctly
initializes the charset of the result string.
into dl145s.mysql.com:/users/mhansson/mysql/push/bug23856/mysql-5.1o-pushee
sql/item_sum.cc:
Auto merged
sql/sql_select.cc:
Auto merged
mysql-test/r/func_gconcat.result:
Bug#23856: Manually merged test case.
mysql-test/t/func_gconcat.test:
Bug#23856: Manually merged test case.
into dl145s.mysql.com:/users/mhansson/mysql/push/bug23856/my51-bug23856
mysql-test/r/func_gconcat.result:
Auto merged
mysql-test/t/func_gconcat.test:
Auto merged
sql/field.h:
Auto merged
sql/field_conv.cc:
Auto merged
sql/sql_select.cc:
Auto merged
sql/item_sum.cc:
Bug#23856: Manual merge 5.0->5.1. Changed comparison to strict inequality between
convert_blob_length and UINT_MAX16. Replaced UINT_MAX16 with Field_varstring::MAX_SIZE.
into mysql.com:/d2/hf/mrg/mysql-5.1-opt
mysql-test/r/func_gconcat.result:
Auto merged
mysql-test/include/mix1.inc:
Auto merged
mysql-test/r/func_group.result:
Auto merged
mysql-test/r/innodb_mysql.result:
Auto merged
mysql-test/t/func_gconcat.test:
Auto merged
mysql-test/t/func_group.test:
Auto merged
sql/item.cc:
Auto merged
sql/item_cmpfunc.cc:
Auto merged
sql/item_sum.cc:
Auto merged
sql/sql_select.cc:
Auto merged
sql/sql_union.cc:
Auto merged
mysql-test/r/ps.result:
merging
mysql-test/r/subselect.result:
merging
mysql-test/r/type_datetime.result:
SCCS merged
mysql-test/t/ps.test:
merging
mysql-test/t/subselect.test:
merging
mysql-test/t/type_datetime.test:
merging
sql/opt_sum.cc:
SCCS merged
When using GROUP_CONCAT with ORDER BY, a tree is used for the sorting, as
opposed to normal nested loops join used when there is no ORDER BY.
The tree traversal that generates the result counts the lines that have been
cut down. (as they get cut down to the field's max_size)
But the check of that count was before the tree traversal, so no
warning was generated if the output is truncated.
Fixed by moving the check to after the tree traversal.
mysql-test/r/func_gconcat.result:
bug#28273: correct result
mysql-test/t/func_gconcat.test:
bug#28273: test case
sql/item_sum.cc:
bug#28273: the fix
Moved the code that outputs a warning to after temporary table (tree) is traversed.
into moonbone.local:/mnt/gentoo64/work/bk-trees/mysql-5.1-opt
mysql-test/r/func_gconcat.result:
Auto merged
mysql-test/r/insert_update.result:
Auto merged
mysql-test/r/subselect3.result:
Auto merged
mysql-test/t/func_gconcat.test:
Auto merged
mysql-test/t/insert_update.test:
Auto merged
sql/field_conv.cc:
Auto merged
sql/item.cc:
Auto merged
sql/sql_base.cc:
Auto merged
sql/sql_load.cc:
Auto merged
sql/sql_select.cc:
Auto merged
sql/table.h:
Auto merged
mysql-test/r/gis.result:
Manual merge
mysql-test/t/gis.test:
Manual merge
sql/handler.cc:
Manual merge
sql/item_sum.cc:
Manual merge
sql/sql_insert.cc:
Manual merge
sql/sql_table.cc:
Manual merge
When creating a temporary table the concise column type
of a string expression is decided based on its length:
- if its length is under 512 it is stored as either
varchar or char.
- otherwise it is stored as a BLOB.
There is a flag (convert_blob_length) to create_tmp_field
that, when >0 allows to force creation of a varchar if the
max blob length is under convert_blob_length.
However it must be verified that convert_blob_length
(settable through a SQL option in some cases) is
under the maximum that can be stored in a varchar column.
While performing that check for expressions in
create_tmp_field_from_item the max length of the blob was
used instead. This causes blob columns to be created in the
heap temp table used by GROUP_CONCAT (where blobs must not
be created in the temp table because of the constant
convert_blob_length that is passed to create_tmp_field() ).
And since these blob columns are not expected in that place
we get wrong results.
Fixed by checking that the value of the flag variable is
in the limits that fit into VARCHAR instead of the max length
of the blob column.
mysql-test/r/func_gconcat.result:
Bug #26815: test case
mysql-test/t/func_gconcat.test:
Bug #26815: test case
sql/item_sum.cc:
Bug #26815: wrong length was checked
sql/sql_select.cc:
Bug #26815: wrong length was checked
