This should also fix the MariaDB 10.2.2 bug
MDEV-13826 CREATE FULLTEXT INDEX on encrypted table fails.
MDEV-12634 FIXME: Modify innodb-index-online, innodb-table-online
so that they will write and read merge sort files. InnoDB 5.7
introduced some optimizations to avoid using the files for small tables.
Many collation test results have been adjusted for MDEV-10191.
The problem was introduced by the patch for MDEV-7661,
which (in addition to the fix itself) included an attempt to make
CONVERT/CAST work in the same way with fields
(i.e. return NULL in strict mode if a non-convertable character found).
It appeared to be a bad idea and some users were affected by this
behavior change. Changing CONVERT/CAST not depend on sql_mode
(restoring pre-10.1.4 behavior).
"Optimization for equi-joins of derived tables with GROUP BY"
should be considered rather as a 'proof of concept'.
The task itself is targeted at an optimization that employs re-writing
equi-joins with grouping derived tables / views into lateral
derived tables. Here's an example of such transformation:
select t1.a,t.max,t.min
from t1 [left] join
(select a, max(t2.b) max, min(t2.b) min from t2
group by t2.a) as t
on t1.a=t.a;
=>
select t1.a,tl.max,tl.min
from t1 [left] join
lateral (select a, max(t2.b) max, min(t2.b) min from t2
where t1.a=t2.a) as t
on 1=1;
The transformation pushes the equi-join condition t1.a=t.a into the
derived table making it dependent on table t1. It means that for
every row from t1 a new derived table must be filled out. However
the size of any of these derived tables is just a fraction of the
original derived table t. One could say that transformation 'splits'
the rows used for the GROUP BY operation into separate groups
performing aggregation for a group only in the case when there is
a match for the current row of t1.
Apparently the transformation may produce a query with a better
performance only in the case when
- the GROUP BY list refers only to fields returned by the derived table
- there is an index I on one of the tables T used in FROM list of
the specification of the derived table whose prefix covers the
the fields from the proper beginning of the GROUP BY list or
fields that are equal to those fields.
Whether the result of the re-writing can be executed faster depends
on many factors:
- the size of the original derived table
- the size of the table T
- whether the index I is clustering for table T
- whether the index I fully covers the GROUP BY list.
This patch only tries to improve the chosen execution plan using
this transformation. It tries to do it only when the chosen
plan reaches the derived table by a key whose prefix covers
all the fields of the derived table produced by the fields of
the table T from the GROUP BY list.
The code of the patch does not evaluates the cost of the improved
plan. If certain conditions are met the transformation is applied.
This patch fills in a serious flaw in the
code that supports condition pushdown into
materialized views / derived tables.
If a predicate happened to contain a reference
to a mergeable view / derived table and it does
not depended directly on the target materialized
view / derived table then the predicate was not
considered as a subject to pusdown to this view
/ derived table.
The bug happens because of a combination of unfortunate circumstances:
1. Arguments args[0] and args[2] of Item_func_concat point recursively
(through Item_direct_view_ref's) to the same Item_func_conv_charset.
Both args[0]->args[0]->ref[0] and args[2]->args[0]->ref[0] refer to
this Item_func_conv_charset.
2. When Item_func_concat::args[0]->val_str() is called,
Item_func_conv_charset::val_str() writes its result to
Item_func_conc_charset::tmp_value.
3. Then, for optimization purposes (to avoid copying),
Item_func_substr::val_str() initializes Item_func_substr::tmp_value
to point to the buffer fragment owned by Item_func_conv_charset::tmp_value
Item_func_substr::tmp_value is returned as a result of
Item_func_concat::args[0]->val_str().
4. Due to optimization to avoid memory reallocs,
Item_func_concat::val_str() remembers the result of args[0]->val_str()
in "res" and further uses "res" to collect the return value.
5. When Item_func_concat::args[2]->val_str() is called,
Item_func_conv_charset::tmp_value gets overwritten (see #1),
which effectively overwrites args[0]'s Item_func_substr::tmp_value (see #3),
which effectively overwrites "res" (see #4).
This patch does the following:
a. Changes Item_func_conv_charset::val_str(String *str) to use
tmp_value and str the other way around. After this change tmp_value
is used to store a temporary result, while str is used to return the value.
The fixes the second problem (without SUBSTR):
SELECT CONCAT(t2,'-',t2) c2
FROM (SELECT CONVERT(t USING latin1) t2 FROM t1) sub;
As Item_func_concat::val_str() supplies two different buffers when calling
args[0]->val_str() and args[2]->val_str(), in the new reduction the result
created during args[0]->val_str() does not get overwritten by
args[2]->val_str().
b. Fixing the same problem in val_str() for similar classes
Item_func_to_base64
Item_func_from_base64
Item_func_weight_string
Item_func_hex
Item_func_unhex
Item_func_quote
Item_func_compress
Item_func_uncompress
Item_func_des_encrypt
Item_func_des_decrypt
Item_func_conv_charset
Item_func_reverse
Item_func_soundex
Item_func_aes_encrypt
Item_func_aes_decrypt
Item_func_buffer
c. Fixing Item_func::val_str_from_val_str_ascii() the same way.
Now Item_str_ascii_func::ascii_buff is used for temporary value,
while the parameter passed to val_str() is used to return the result.
This fixes the same problem when conversion (from ASCII to e.g. UCS2)
takes place. See the ctype_ucs.test for example queries that returned
wrong results before the fix.
d. Some Item_func descendand classes had temporary String buffers
(tmp_value and tmp_str), but did not really use them.
Removing these temporary buffers from:
Item_func_decode_histogram
Item_func_format
Item_func_binlog_gtid_pos
Item_func_spatial_collection:
e. Removing Item_func_buffer::tmp_value, because it's not used any more.
f. Renaming Item_func_[un]compress::buffer to "tmp_value",
for consistency with other classes.
Note, this patch does not fix the following classes
(although they have a similar problem):
Item_str_conv
Item_func_make_set
Item_char_typecast
They have a complex implementations and simple swapping between "tmp_value"
and "str" won't work. These classes will be fixed separately.
The problem lies in how CURRENT_ROLE is defined. The
Item_func_current_role inherits from Item_func_sysconst, which defines
a safe_charset_converter to be a const_charset_converter.
During view creation, if there is no role previously set, the current_role()
function returns NULL.
This is captured on item instantiation and the
const_charset_converter call subsequently returns an Item_null.
In turn, the function is replaced with Item_null and the view is
then created with an Item_null instead of Item_func_current_role.
Without this patch, the first SHOW CREATE VIEW from the testcase would
have a where clause of WHERE role_name = NULL, while the second SHOW
CREATE VIEW would show a correctly created view.
The same applies for the DATABASE function, as it can change as well.
There is an additional problem with CURRENT_ROLE() when used in a
prepared statement. During prepared statement creation we used to set
the string_value of the function to the current role as well as the
null_value flag. During execution, if CURRENT_ROLE was not null, the
null_value flag was never set to not-null during fix_fields.
Item_func_current_user however can never be NULL so it did not show this
problem in a view before. At the same time, the CURRENT_USER() can not
be changed between prepared statement execution and creation so the
implementation where the value is stored during fix_fields is
sufficient.
Note also that DATABASE() function behaves differently during prepared
statements. See bug 25843 for details or commit
7e0ad09edf
sql_sequence.read_only: Show that the sequence can be read in
both read-only and read-write mode, and that the sequence remains
accessible after a server restart.
This is a joint patch fixing the following problems:
MDEV-12875 Wrong VIEW column data type for COALESCE(int_column)
MDEV-12886 Different default for INT and BIGINT column in a VIEW for a SELECT with ROLLUP
MDEV-12916 Wrong column data type for an INT field of a cursor-anchored ROW variable
All above problem happened because the global function ::create_tmp_field()
called the top-level Item::create_tmp_field(), which made some tranformation
for INT-result data types. For example, INT(11) became BIGINT(11), because 11
is a corner case and it's not known if it fits or does not fit into INT range,
so Item::create_tmp_field() converted it to BIGINT(11) for safety.
The main idea of this patch is to avoid such tranformations.
1. Fixing Item::create_tmp_field() not to have a special case for INT_RESULT.
Item::create_tmp_field() is changed not to have a special case
for INT_RESULT (which earlier made a decision based on Item's max_length).
It now calls tmp_table_field_from_field_type() for INT_RESULT,
therefore preserves the original data type (e.g. INT, YEAR) without
conversion to BIGINT.
This change is valid, because a number of recent fixes
(e.g. in Item_func_int, Item_hybrid_func, Item_int, Item_splocal)
guarantee that item->type_handler() now properly returns
type_handler_long vs type_handler_longlong. So no adjustment by length
is needed any more for Items returning INT_RESULT.
After this change, Item::create_tmp_field() calls
tmp_table_field_from_field_type() for all XXX_RESULT, except REAL_RESULT.
2. Fixing Item::create_tmp_field() not to have a special case for REAL_RESULT.
Note, the reason for a special case for REAL_RESULT is to have a special
constructor for Field_double(), forcing Field_real::not_fixed to be set
to true.
Taking into account that only Item_sum descendants actually need a special
constructor call Field_double(not_fixed=true), not too loose precision
when mixing individual rows to the aggregate result:
- renaming Item::create_tmp_field() to Item_sum::create_tmp_field().
- changing Item::create_tmp_field() just to call
tmp_table_field_from_field_type() for all XXX_RESULT types.
A special case for REAL_RESULT in Item::create_tmp_field() is now gone.
Item::create_tmp_field() is now symmetric for all XXX_RESULT types,
and now just calls tmp_table_field_from_field_type().
3. Fixing Item_func::create_field_for_create_select() not to have
a special case for STRING_RESULT.
After changes #1 and #2, the code in
Item_func::create_field_for_create_select(), testing result_type(),
becomes useless, because: now Item::create_tmp_field() and
tmp_table_field_from_field_type() do exactly the same thing for all
XXX_RESULT types for Item_func descendants:
a. It calls tmp_table_field_from_field_type for STRING_RESULT directly.
b. For other XXX_RESULT, it goes through Item::create_tmp_field(),
which calls the global function ::create_tmp_field(),
which calls item->create_tmp_field() for FUNC_ITEM,
which calls tmp_table_field_from_field_type() again.
So removing the virtual implementation of
Item_func::create_field_for_create_select().
The inherited Item::create_field_for_create_select() now perfectly
does the job, as it also calls tmp_table_field_from_field_type()
for FUNC_ITEM, independently from XXX_RESULT type.
4. Taking into account #1 and #2, as well as some recent changes,
removing virtual implementations:
- Item_hybrid_func::create_tmp_field()
- Item_hybrid_func::create_field_for_create_select()
- Item_int_func::create_tmp_field()
- Item_int_func::create_field_for_create_select()
- Item_temporal_func::create_field_for_create_select()
The derived versions from Item now perfectly work.
5. Moving a piece of code from create_tmp_field_from_item()
to a new function create_tmp_field_from_item_finalize(),
to reuse it in two places (see #6).
6. Changing the code responsible for BIT->INT/BIGIN tranformation
(which is called for the cases when the created table, e.g. HEAP,
does not fully support BIT) not to call create_tmp_field_from_item(),
because the latter now calls tmp_table_field_from_field_type() instead
of create_tmp_field() and thefore cannot do BIT transformation.
So rewriting this code using a sequence of these calls:
- item->type_handler_long_or_longlong()
- handler->make_and_init_table_field()
- create_tmp_field_from_item_finalize()
7. Miscelaneous changes:
- Moving type_handler_long_or_longlong() from "protected" to "public",
as it's now needed in the global function create_tmp_field().
8. The above changes fixed MDEV-12875, MDEV-12886, MDEV-12916.
So adding tests for these bugs.
This is a joint patch for:
MDEV-12852 Out-of-range errors when CAST(1-2 AS UNSIGNED
MDEV-12853 Out-of-range errors when CAST('-1' AS UNSIGNED
MDEV-12869 Wrong metadata for integer additive and multiplicative operators
1. Fixing all Item_func_numhybrid descendants to set the precise
data type handler (type_handler_long or type_handler_longlong)
at fix_fields() time. This fixes MDEV-12869.
2. Fixing Item_func_unsigned_typecast to set the precise data type handler
at fix_fields() time. This fixes MDEV-12852 and MDEV-12853.
This is done by:
- fixing Type_handler::Item_func_unsigned_fix_length_and_dec()
and Type_handler_string_result::Item_func_unsigned_fix_length_and_dec()
to properly detect situations when a negative epxression is converted
to UNSIGNED. In this case, length of the result is now always set to
MAX_BIGINT_WIDTH without trying to use args[0]->max_length, as very
short arguments can produce very long result in such conversion:
CAST(-1 AS UNSIGNED) -> 18446744073709551614
- adding a new virtual method "longlong Item::val_int_max() const",
to preserve the old behavior for expressions like this:
CAST(1 AS UNSIGNED)
to stay under the INT data type (instead of BIGINT) for small
positive integer literals. Using Item::unsigned_flag would not help,
because Item_int does not set unsigned_flag to "true" for positive
numbers.
3. Adding helper methods:
* Item::type_handler_long_or_longlong()
* Type_handler::type_handler_long_or_longlong()
and reusing them in a few places, to reduce code duplication.
4. Making reorganation in create_tmp_field() and
create_field_for_create_select() for Item_hybrid_func and descendants,
to reduce duplicate code. They all now have a similar behavior in
respect of creating fields. Only Item_func_user_var descendants have
a different behavior. So moving the default behvior to Item_hybrid_func,
and overriding behavior on Item_func_user_var level.
MDEV-12858 Out-of-range error for CREATE..SELECT unsigned_int_column+1
MDEV-12859 Out-of-range error for CREATE..SELECT @a:=EXTRACT(MINUTE_MICROSECOND FROM..)
MDEV-12862 Data type of @a:=1e0 depends on the session character set
1. Moving a part of Item::create_tmp_field() into a new helper method
Item::create_tmp_field_int() and reusing it in Item::create_tmp_field()
and Item_func_signed::create_tmp_field().
Fixing the code in Item::create_tmp_field_int() to call
Type_handler::make_table_field() instead of doing "new Field_long[long]"
directly. This change revealed a problem reported in MDEV-12862.
2. Changing the "long vs longlong" cut-off length for
- Item_func::create_tmp_field()
- Item_sum::create_tmp_field()
- Item_func_get_user_var::create_tmp_field()
from MY_INT32_NUM_DECIMAL_DIGITS to (MY_INT32_NUM_DECIMAL_DIGITS - 2).
This fixes MDEV-12858.
After this change, the "convert_int_length" parameter to
Item::create_tmp_field() is not needed any more, because
(MY_INT32_NUM_DECIMAL_DIGITS - 2) is always passed.
So removing the "convert_int_length" parameter.
3. Fixing Item::create_tmp_field() to pass max_char_length() instead
of max_length to the constructor of Field_double().
This fixes MDEV-12862.
4. Additionally, fixing
- Type_handler_{tiny|short|int24|long|longlong}::make_table_field()
- Type_handler_{float|double}::make_table_field()
to pass max_char_length() instead of max_length to Field contructors.
This is needed by the change (1).
5. Adding new tests, and recording new correct results in the old tests in:
- mysql-test/r/type_ranges.result
- storage/tokudb/mysql-test/tokudb/r/type_ranges.result
Introducing a new class Type_holder (used internally in sql_union.cc),
to reuse exactly the same data type attribute aggregation Type_handler API
for hybrid functions and UNION.
This fixes a number of bugs in UNION:
- MDEV-9495 Wrong field type for a UNION of a signed and an unsigned INT expression
- MDEV-9497 UNION and COALESCE produce different field types for DECIMAL+INT
- MDEV-12594 UNION between fixed length double columns does not always preserve scale
- MDEV-12595 UNION converts INT to BIGINT
- MDEV-12599 UNION is not symmetric when mixing INT and CHAR
Details:
- sql_union.cc: Reusing attribute aggregation for UNION.
Adding new methods:
* st_select_lex_unit::join_union_type_handlers()
* st_select_lex_unit::join_union_type_attributes()
* st_select_lex_unit::join_union_item_types()
Removing the old join_types()-based code.
- Changing Type_handler::Item_hybrid_func_fix_attributes()
to accept "name", Type_handler_hybrid_field_type, Type_all_attributes
as three separate parameters instead of a single Item_hybrid_func parameter,
to make it possible to pass both Item_hybrid_func and Type_holder.
- Moving the former special GEOMETRY and ENUM/SET attribute aggregation code
from Item_type_holder::join_types() to
* Type_handler_typelib::Item_hybrid_func_fix_attributes().
* Type_handler_geometry::Item_hybrid_func_fix_attrubutes().
This makes GEOMETRY/ENUM/SET symmetric with all other data types
(from the UNION point of view).
Removing Item_type_holder::join_types() and Item_type_holder::get_full_info().
- Adding new methods into Type_all_attributes:
* Type_all_attributes::set_geometry_type() and
Item_hybrid_func::set_geometry_type().
* Adding Type_all_attributes::get_typelib().
* Adding Type_all_attributes::set_typelib().
- Adding Type_handler_typelib as a common parent for
Type_handler_enum and Type_handler_set, to avoid code duplication: they have
already had two common methods, and we're adding one more shared method.
- Adding Type_all_attributes::set_maybe_null(), as some type handlers
may want to set maybe_null (e.g. Type_handler_geometry) during data type
attribute aggregation.
- Changing Type_geometry_attributes() to accept Type_handler
and Type_all_attributes as two separate parameters, instead
of a single Item parameter, to make it possible to pass Type_holder.
- Adding Item_args::add_argument().
- Moving Item_args::alloc_arguments() from "protected" to "public".
- Moving Item_type_holder::Item_type_holder() from item.cc to item.h, as
now it's very simple.
Btw, this constructor should probably be eventually removed.
It's now used only in sql_show.cc, which could be modified to use
Item_return_decimal (for symmetry with Item_return_xxx created for all
other data types). Or, another option: remove all Item_return_xxx and
use Item_type_holder for all data types instead.
- storage/tokudb/mysql-test/tokudb/r/type_float.result
Recording new results (MDEV-12594).
- mysql-test/r/cte_recursive.result
Recording new results (MDEV-9497)
- mysql-test/r/subselect*.result
Recording new results (MDEV-12595)
- mysql-test/r/metadata.result
Recording new results (MDEV-9495)
- mysql-test/r/temp_table.result
Recording new results (MDEV-12594)
- mysql-test/r/type_float.result
Recording new results (MDEV-12594)
This is a join patch fixing these two bugs:
MDEV-12560 Wrong data type for SELECT NULL UNION SELECT Point(1,1)
MDEV-12665 Hybrid functions do not preserve geometry type
