The bug caused a memory overwrite in the function update_ref_and_keys()
It happened due to a wrong value of SELECT_LEX::cond_count. This value
historically was calculated by the fix_fields method. Now the logic of
calling this method became too complicated and, as a result, this value
is calculated not always correctly.
The patch changes the way how and when the values of SELECT_LEX::cond_count
and of SELECT_LEX::between_count are calculated. The new code does it just at
the beginning of update_ref_and_keys().
Other fix of maybe_null problem and revert of revno: 3608 "MDEV-3873 & MDEV-3876 & MDEV-3912 : Wrong result (extra rows) with ALL subquery from a MERGE view."
STRING CONVERSION FUNCTIONS
Problem:
While executing the prepared statement, user variable is
set to memory which would be freed at the end of
execution.
If the statement is executed again, valgrind throws
error when accessing this pointer.
Analysis:
1. First time when Item_func_set_user_var::check is called,
memory is allocated for "value" to store the result.
(In the call to copy_if_not_alloced).
2. While sending the result, Item_func_set_user_var::check
is called again. But, this time, its called with
"use_result_field" set to true.
As a result, we call result_field->val_str(&value).
3. Here memory allocated for "value" gets freed. And "value"
gets set to "result_field", with "str_length" being that of
result_field's.
4. In the call to JOIN::cleanup, result_field's memory gets
freed as this is allocated in a chunk as part of the
temporary table which is needed to execute the query.
5. Next time, when execute of the same statement is called,
"value" will be set to memory which is already freed.
Valgrind error occurs as "str_length" is positive
(set at Step 3)
Note that user variables list is stored as part of the Lex object
in set_var_list. Hence the persistance across executions.
Solution:
Patch for Bug#11764371 fixed in mysql-5.6+ fixes this problem
as well.So backporting the same.
In the solution for Bug#11764371, we create another object of
user_var and repoint it to temp_table's field. As a result while
deleting the alloced buffer in Step 3, since the cloned object
does not own the buffer, deletion will not happen.
So at step 5 when we execute the statement second time, the
original object will be used and since deletion did not happen
valgrind will not complain about dangling pointer.
sql/item_func.h:
Add constructors.
sql/sql_select.cc:
Change user variable assignment functions to read from fields after
tables have been unlocked.
STRING CONVERSION FUNCTIONS
Problem:
While executing the prepared statement, user variable is
set to memory which would be freed at the end of
execution.
If the statement is executed again, valgrind throws
error when accessing this pointer.
Analysis:
1. First time when Item_func_set_user_var::check is called,
memory is allocated for "value" to store the result.
(In the call to copy_if_not_alloced).
2. While sending the result, Item_func_set_user_var::check
is called again. But, this time, its called with
"use_result_field" set to true.
As a result, we call result_field->val_str(&value).
3. Here memory allocated for "value" gets freed. And "value"
gets set to "result_field", with "str_length" being that of
result_field's.
4. In the call to JOIN::cleanup, result_field's memory gets
freed as this is allocated in a chunk as part of the
temporary table which is needed to execute the query.
5. Next time, when execute of the same statement is called,
"value" will be set to memory which is already freed.
Valgrind error occurs as "str_length" is positive
(set at Step 3)
Note that user variables list is stored as part of the Lex object
in set_var_list. Hence the persistance across executions.
Solution:
Patch for Bug#11764371 fixed in mysql-5.6+ fixes this problem
as well.So backporting the same.
In the solution for Bug#11764371, we create another object of
user_var and repoint it to temp_table's field. As a result while
deleting the alloced buffer in Step 3, since the cloned object
does not own the buffer, deletion will not happen.
So at step 5 when we execute the statement second time, the
original object will be used and since deletion did not happen
valgrind will not complain about dangling pointer.
sql/item_func.h:
Add constructors.
sql/sql_select.cc:
Change user variable assignment functions to read from fields after
tables have been unlocked.
with decimals=NOT_FIXED_DEC it is possible to have 'decimals' larger
than 'max_length', it's not an error for temporal functions.
But when Item_func_numhybrid converts the value to DECIMAL_RESULT,
it must limit 'decimals' to be a valid scale of a decimal number.
Backport of Bug#13581962
mysql-test/r/cast.result:
Added test result for Bug#13581962,Bug#14096619
mysql-test/t/cast.test:
Added test case for Bug#13581962,Bug#14096619
sql/item_func.h:
limit max length by MY_INT64_NUM_DECIMAL_DIGITS
Analysis
The reason for the less efficient plan was result of a prior design decision -
to limit the eveluation of constant expressions during optimization to only
non-expensive ones. With this approach all stored procedures were considered
expensive, and were not evaluated during optimization. As a result, SPs didn't
participate in range optimization, which resulted in a plan with table scan
rather than index range scan.
Solution
Instead of considering all SPs expensive, consider expensive only those SPs
that are non-deterministic. If an SP is deterministic, the optimizer will
checj if it is constant, and may eventually evaluate it during optimization.
from a MERGE view.
The problem was in the lost ability to be null for the table of a left join if it
is a view/derived table.
It hapenned because setup_table_map(), was called earlier then we merged
the view or derived.
Fixed by propagating new maybe_null flag during Item::update_used_tables().
Change in join_outer.test and join_outer_jcl6.test appeared because
IS NULL reported no used tables (i.e. constant) for argument which could not be
NULL and new maybe_null flag was propagated for IS NULL argument (Item_field)
because table the Item_field belonged to changed its maybe_null status.
Problem:-
using last_insert_id() on an auto_incremented bigint unsigned does
not work for values which are greater than max-bigint-signed.
Analysis:-
last_insert_id() returns the first auto_incremented value for a column
and an auto_incremented value can have only positive values.
In our code, when we are initializing a last_insert_id object, we are
taking it as a signed BIGINT, So when the auto_incremented value reaches
greater than max signed bigint, last_insert_id gives negative result.
Solution:
When we are fetching the value from last_insert_id, We are setting the
unsigned_flag, so that it take only unsigned BIGINT value.
sql/item_func.cc:
here unsigned value is converted to signed value.
sql/item_func.h:
last_insert_id() gives an auto_incremented value which can be
positive only,so defined it as a unsigned longlong sets the
unsigned_flag to 1.
- The problem was that const-table-reading code would try to evaluate MATCH()
before init_ftfuncs() was called.
- Fixed by making MATCH function "expensive" so that nobody tries to evaluate it
at optimization phase.
Bug#11766642: crash in Item_field::register_field_in_read_map
with view
(Former 59793)
Prior to the refactoring in this patch, Item_cond_xor behaved
partially as an Item_cond and partially as an Item_func. The
reasoning behind this was that XOR is currently not optimized
(thus should be Item_func instead of Item_cond), but it was
planned optimize it in the future (thus, made Item_cond anyway
to ease optimization later).
Even though Item_cond inherits from Item_func, there are
differences between these two. One difference is that the
arguments are stored differently. Item_cond stores them in a
list while Item_func store them in an args[].
BUG no 45221 was caused by Item_cond_xor storing arguments in
the list while users of the objects would look for them in
args[]. The fix back then was to store the arguments in both
locations.
In this bug, Item_cond_xor initially gets two Item_field
arguments. These are stored in the list inherited from
Item_cond and in args[] inherited from Item_func. During
resolution, find_field_in_view() replaces the Item_fields
stored in the list with Item_direct_view_refs, but args[]
still points to the unresolved Item_fields. This shows that
the fix for 45221 was incorrect.
The refactoring performed in this patch removes the confusion
by making the XOR item an Item_func period. A neg_transformer()
is also implemented for Item_func_xor to improve performance
when negating XOR expressions. An XOR is negated by negating
one of the operands.
A lot of small fixes and new test cases.
client/mysqlbinlog.cc:
Cast removed
client/mysqltest.cc:
Added missing DBUG_RETURN
include/my_pthread.h:
set_timespec_time_nsec() now only takes one argument
mysql-test/t/date_formats.test:
Remove --disable_ps_protocl as now also ps supports microseconds
mysys/my_uuid.c:
Changed to use my_interval_timer() instead of my_getsystime()
mysys/waiting_threads.c:
Changed to use my_hrtime()
sql/field.h:
Added bool special_const_compare() for fields that may convert values before compare (like year)
sql/field_conv.cc:
Added test to get optimal copying of identical temporal values.
sql/item.cc:
Return that item_int is equal if it's positive, even if unsigned flag is different.
Fixed Item_cache_str::save_in_field() to have identical null check as other similar functions
Added proper NULL check to Item_cache_int::save_in_field()
sql/item_cmpfunc.cc:
Don't call convert_constant_item() if there is nothing that is worth converting.
Simplified test when years should be converted
sql/item_sum.cc:
Mark cache values in Item_sum_hybrid as not constants to ensure they are not replaced by other cache values in compare_datetime()
sql/item_timefunc.cc:
Changed sec_to_time() to take a my_decimal argument to ensure we don't loose any sub seconds.
Added Item_temporal_func::get_time() (This simplifies some things)
sql/mysql_priv.h:
Added Lazy_string_decimal()
sql/mysqld.cc:
Added my_decimal constants max_seconds_for_time_type, time_second_part_factor
sql/table.cc:
Changed expr_arena to be of type CONVENTIONAL_EXECUTION to ensure that we don't loose any items that are created by fix_fields()
sql/tztime.cc:
TIME_to_gmt_sec() now sets *in_dst_time_gap in case of errors
This is needed to be able to detect if timestamp is 0
storage/maria/lockman.c:
Changed from my_getsystime() to set_timespec_time_nsec()
storage/maria/ma_loghandler.c:
Changed from my_getsystime() to my_hrtime()
storage/maria/ma_recovery.c:
Changed from my_getsystime() to mmicrosecond_interval_timer()
storage/maria/unittest/trnman-t.c:
Changed from my_getsystime() to mmicrosecond_interval_timer()
storage/xtradb/handler/ha_innodb.cc:
Added support for new time,datetime and timestamp
unittest/mysys/thr_template.c:
my_getsystime() -> my_interval_timer()
unittest/mysys/waiting_threads-t.c:
my_getsystime() -> my_interval_timer()
Fields belonging to views in general cannot be substituted for
equal items, in particular for constants, because all references
to a view field refer to the same Item_field object while they
could be used in different OR parts of the where condition and
belong to different equivalence classes (to different Item_equals).
That's why substitution for equal items in any context is allowed
only in place of Item_direct_view_ref objects, but not in place of
Item_fields these objects refer to.
Due to some erroneous code in the patch for bug 717577 substitution
for view fields were allowed in some context.This could lead
to wrong results returned by queries using views.
The fix prohibits substitution of view fields for equal items
in any context.
The patch also changes slightly the compile method for the Item_func
class. Now if the analyze method returns NULL in his parameter the
compile method is not called for the arguments of the function
at all. A similar change was made for the Item_ref class.
Resolved all conflicts, bad merges and fixed a few minor bugs in the code.
Commented out the queries from multi_update, view, subselect_sj, func_str,
derived_view, view_grant that failed either with crashes in ps-protocol or
with wrong results.
The failures are clear indications of some bugs in the code and these bugs
are to be fixed.
- COLUMN_CREATE(column_nr, value, [column_nr,value]...)
- COLUMN_ADD(blob,column_nr, value, column_nr,value]...)
- COLUMN_DELETE(blob, column_nr, column_nr...)
- COLUMN_EXISTS(blob, column_nr)
- COLUMN_LIST(blob, column_nr)
- COLUMN_GET(string, column_nr AS type)
Added cast(X as DOUBLE) and cast(x as INT)
Better warning and error messages for wrong cast's
Created some sub functions to simplify and reuse code.
Added a lot of conversation functions with error/warnings for what went wrong.
Fixed some issues when casting time to datetime.
Added functions to dynamic strings and Strings to allow one to move a string buffer from dynamic strings to String (to save malloc+ copy)
Added dynamic columns library to libmysqlclient
include/Makefile.am:
Added ma_dyncol.h
include/decimal.h:
Added 'const' to arguments for some functions.
include/my_sys.h:
Added dynstr_reassociate()
include/my_time.h:
Added TIME_SUBSECOND_RANGE
Added double_to_datetime()
Added flag argument to str_to_time()
libmysql/CMakeLists.txt:
Added mysys/ma_dyncol.c
libmysql/Makefile.shared:
Added ma_dyncol
libmysql/libmysql.c:
Added argument to str_to_time()
mysql-test/r/bigint.result:
Better error messages
mysql-test/r/cast.result:
Better warning and error messages
A lot of new cast() tests
mysql-test/r/func_math.result:
Better warning messages
mysql-test/r/func_str.result:
Better warning messages
mysql-test/r/func_time.result:
Better warning messages
mysql-test/r/sp-vars.result:
Better warning messages
mysql-test/r/strict.result:
Better warning messages
New test result
mysql-test/r/type_newdecimal.result:
Better warning messages
mysql-test/r/warnings.result:
Better warning messages
mysql-test/suite/funcs_1/r/innodb_func_view.result:
Updated results after better cast warnings
mysql-test/suite/funcs_1/r/memory_func_view.result:
Updated results after better cast warnings
mysql-test/suite/funcs_1/r/myisam_func_view.result:
Updated results after better cast warnings
mysql-test/suite/optimizer_unfixed_bugs/t/bug43448.test:
Added begin...commit to speed up test.
mysql-test/suite/parts/inc/part_supported_sql_funcs_delete.inc:
Added begin...commit to speed up test.
mysql-test/suite/parts/inc/partition_supported_sql_funcs.inc:
Added begin...commit to speed up test.
mysql-test/suite/parts/r/part_supported_sql_func_innodb.result:
Added begin...commit to speed up test.
mysql-test/suite/parts/r/part_supported_sql_func_myisam.result:
Added begin...commit to speed up test.
mysql-test/suite/parts/r/rpl_partition.result:
Added begin...commit to speed up test.
mysql-test/suite/parts/t/part_supported_sql_func_innodb.test:
Removed duplicated --big_test
mysql-test/suite/parts/t/rpl_partition.test:
Added begin...commit to speed up test.
mysql-test/suite/pbxt/r/cast.result:
Updated results after better cast warnings
mysql-test/suite/pbxt/r/func_str.result:
Updated results after better cast warnings
mysql-test/suite/pbxt/r/type_newdecimal.result:
Updated results after better cast warnings
mysql-test/suite/rpl/r/rpl_innodb_bug28430.result:
Added begin...commit to speed up test.
mysql-test/suite/rpl/t/rpl_innodb_bug28430.test:
Added begin...commit to speed up test.
mysql-test/suite/vcol/r/vcol_supported_sql_funcs_innodb.result:
More warnings
mysql-test/suite/vcol/r/vcol_supported_sql_funcs_myisam.result:
More warnings
mysql-test/t/cast.test:
A lot of new cast() tests
mysql-test/t/strict.test:
Added new test
mysys/CMakeLists.txt:
Added ma_dyncol.c
mysys/Makefile.am:
Added ma_dyncol.c
mysys/string.c:
Added dynstr_reassociate() to move a buffer from dynamic_strings to some other allocator
sql-common/my_time.c:
Added 'fuzzydate' flag to str_to_time()
Added support for microseconds to my_time_to_str() and my_datetime_to_str()
Reset second_parts in number_to_datetime()
Added double_to_datetime()
sql/field.cc:
Added double_to_longlong() and truncate_double() to simplify and reuse code
sql/field.h:
New prototypes
sql/item.cc:
Changed Item::get_date(MYSQL_TIME *ltime,uint fuzzydate) to be aware of type of argument.
(Needed to make it microsecond safe and get better warnings).
Updated call to str_to_time_with_warn()
sql/item.h:
Added struct st_dyncall_create_def used by dynamic columns
Added virtual bool dynamic_result() to tell if type of argument may change over calls.
sql/item_cmpfunc.cc:
Added Item_func_dyncol_exists()
sql/item_cmpfunc.h:
Added class Item_func_dyncol_exists
sql/item_create.cc:
Added get_length_and_scale() to simplify other functions
Simplified and extended create_func_cast()
Added support for cast(X as double(X,Y))
Added functions to create dynamic column functions.
sql/item_create.h:
Added prototypes
sql/item_func.cc:
Extended cast functions Item_func_signed() and Item_func_unsigned() to work with dynamic types
Added Item_double_typecast()
sql/item_func.h:
Added class Item_double_typecast()
sql/item_strfunc.cc:
Added functions for COLUMN_CREATE(), COLUMN_ADD(), COLUMN_GET() and COLUMN_LIST()
sql/item_strfunc.h:
Added classes for COLUMN_CREATE(), COLUMN_ADD(), COLUMN_GET() and COLUMN_LIST()
sql/item_timefunc.cc:
Added flag argument to str_to_time_with_warn()
Updated Item_char_typecast() to handle result type that may change between calls (for dynamic columns)
Added Item_time_typecast::get_date() to ensure that we cast a datetime to time properly.
sql/item_timefunc.h:
Added get_date() to Item_time_typecast() to allow proper results for casting time to datetime
sql/lex.h:
Added new SQL function names
sql/my_decimal.cc:
Added 'const' to some arguments.
Better error message in case of errors (we now print out the wrong value)
Added my_decimal2int()
sql/my_decimal.h:
Moved some constants to my_decimal_limits.h
Updated prototypes.
Made my_decimal2int() a function as it's rather long (no reason to have it inline)
Added decimal2my_decimal() function.
sql/mysql_priv.h:
Prototypes for new functions
sql/share/errmsg.txt:
New error messages for wrong casts and dynamic columns
sql/sql_acl.cc:
Fixed indentation
sql/sql_base.cc:
Added dynamic_column_error_message()
sql/sql_string.h:
Added reassociate() to move a buffer to be owned by String object.
sql/sql_yacc.yy:
Added syntax for COLUMN_ functions.
sql/time.cc:
Updated str_to_datetime_with_warn() flag argument to same type as other functions
Added conversion flag to str_to_time_with_warn() (Similar to all datetime functions)
Added conversion functions with warnings: double_to_datetime_with_warn() and decimal_to_datetime_with_warn()
strings/decimal.c:
Added 'const' to arguments for some functions.
unittest/mysys/Makefile.am:
Added test for dynamic columns code
Analysis:
A query with implicit grouping is one with aggregate functions and
no GROUP BY clause. MariaDB inherits from MySQL an SQL extenstion
that allows mixing aggregate functions with non-aggregate fields.
If a query with such mixed select clause produces an empty result
set, the meaning of aggregate functions is well defined - either
NULL (MIN, MAX, etc.), or 0 (count(*)). However the non-aggregated
fields must also have some value, and the only reasonable value in
the case of empty result is NULL.
The cause of the many wrong results was that if a field is declared
as non-nullable (e.g. because it is a PK or NOT NULL), the semantic
analysis and the optimization phases treat this field as non-nullable,
and generate all related query plan elements based on this assumption.
Later during execution, these incorrectly configured/generated query
plan elements result in a wrong result because the selected fields
are not null due to the not-null assumption during optimization.
Solution:
Detect before the context analysys phase that a query uses implicit
grouping with mixed aggregates/non-aggregates, and set all fields
as nullable. The parser already walks the SELECT clause, and
already sets Item::with_sum_func for Items that reference aggreagate
functions. The patch adds a symmetric Item::with_field so that all
Items that reference an Item_field are marked during their
construction at parse time in the same way as with aggregate function
use.
