group concat tree is allocated in a memroot, so the only way to free
memory is to copy a part of the tree into a new memroot.
track the accumilated length of the result, and when it crosses
the threshold - copy the result into a new tree, free the old one.
Queries involving rollup need all aggregate function to have copy_or_same function where we create a copy
of item_sum items for each sum level.
Implemented copy_or_same function for the custom aggregate function class (Item_sum_sp)
Problems:
1. Unlike Item_field::fix_fields(),
Item_sum_sp::fix_length_and_dec() and Item_func_sp::fix_length_and_dec()
did not run the code which resided in adjust_max_effective_column_length(),
therefore they did not extend max_length for the integer return data types
from the user-specified length to the maximum length according to
the data type capacity.
2. The code in adjust_max_effective_column_length() was not correct
for TEXT data, because Field_blob::max_display_length()
multiplies to mbmaxlen. So TEXT variants were unintentionally
promoted to the next longer data type for multi-byte character
sets: TINYTEXT->TEXT, TEXT->MEDIUMTEXT, MEDIUMTEXT->LONGTEXT.
3. Item_sum_sp::create_table_field_from_handler()
Item_func_sp::create_table_field_from_handler()
erroneously called tmp_table_field_from_field_type(),
which converted VARCHAR(>512) to TEXT variants.
So "CREATE..SELECT spfunc()" erroneously converted
VARCHAR to TEXT. This was wrong, because stored
functions have explicitly declared data types,
which should be preserved.
Solution:
- Removing Type_std_attributes(const Field *)
and using instead Type_std_attributes::set() in combination
with field->type_str_attributes() all around the code, e.g.:
Type_std_attributes::set(field->type_std_attributes())
These two ways of copying attributes from a Field
to an Item duplicated each other, and were slightly
different in how to mix max_length and mbmaxlen.
- Removing adjust_max_effective_column_length() and
fixing Field::type_std_attributes() to do all necessary
type-specific calculations , so no further adjustments
is needed.
Field::type_std_attributes() is now called from all affected methods:
Item_field::fix_fields()
Item_sum_sp::fix_length_and_dec()
Item_func_sp::fix_length_and_dec()
This fixes the problem N1.
- Making Field::type_std_attributes() virtual, to make
sure that type-specific adjustments a properly done
by individual Field_xxx classes. Implementing
Field_blob::type_std_attributes() in the way that
no TEXT promotion is done.
This fixes the problem N2.
- Fixing Item_sum_sp::create_table_field_from_handler()
Item_func_sp::create_table_field_from_handler() to
call create_table_field_from_handler() instead of
tmp_table_field_from_field_type() to avoid
VARCHAR->TEXT conversion on "CREATE..SELECT spfunc()".
- Recording mysql-test/suite/compat/oracle/r/sp-param.result
as "CREATE..SELECT spfunc()" now correctly
preserve the data type as specified in the RETURNS clause.
- Adding new tests
This commit implements aggregate stored functions. The basic idea behind
the feature is:
* Implement a special instruction FETCH GROUP NEXT ROW that will pause
the execution of the stored function. When the instruction is reached,
execution of the initial query resumes "as if" the function returned.
This gives the server the opportunity to advance to the next row in the
result set.
* Stored aggregates behave like regular aggregate functions. The
implementation of thus resides in the class Item_sum_sp. Because it is
an aggregate function, for each new row in the group, the
Item_sum_sp::add() method will be called. This is when execution resumes
and the function does another iteration to "add" one extra element to
the final result.
* When the end of group is reached, val_xxx() method will be called for
the item. This case is handled by another execute step for the stored
function, only with a special flag to force a call to the return
handler. See Item_sum_sp::execute() for details.
To allow this pause and resume semantic, we must preserve the function
context across executions. This is stored in Item_sp::sp_query_arena only for
aggregate stored functions, but has no impact for regular functions.
We also enforce aggregate functions to include the "FETCH GROUP NEXT ROW"
instruction.
Signed-off-by: Vicențiu Ciorbaru <vicentiu@mariadb.org>
Spider patches 026 (MDEV-7723), 031 (MDEV-7727) and 058 (MDEV-12532)
This allows the storage engine to internally compute sum and count
operations.
- Enhance sum items to be able to store the sum value directly.
- return_record_by_parent() is enabled in spider as
HANDLER_HAS_DIRECT_AGGREGATE is defined
- Added spd_environ.h to spider. This is loaded first to ensure that all
MariaDB specific defines that are used by include files are properly
defined.
- This code is tested by the existing spider tests direct_aggregate.test
and direct_aggregate_part.test and also partition.test
TODO:
- Make get_thd_memroot() inline
- To do this, we need to reduce dependence of include files, especially
so that sql_class.h is not depending in item.h
This is a joint patch fixing the following problems:
MDEV-12875 Wrong VIEW column data type for COALESCE(int_column)
MDEV-12886 Different default for INT and BIGINT column in a VIEW for a SELECT with ROLLUP
MDEV-12916 Wrong column data type for an INT field of a cursor-anchored ROW variable
All above problem happened because the global function ::create_tmp_field()
called the top-level Item::create_tmp_field(), which made some tranformation
for INT-result data types. For example, INT(11) became BIGINT(11), because 11
is a corner case and it's not known if it fits or does not fit into INT range,
so Item::create_tmp_field() converted it to BIGINT(11) for safety.
The main idea of this patch is to avoid such tranformations.
1. Fixing Item::create_tmp_field() not to have a special case for INT_RESULT.
Item::create_tmp_field() is changed not to have a special case
for INT_RESULT (which earlier made a decision based on Item's max_length).
It now calls tmp_table_field_from_field_type() for INT_RESULT,
therefore preserves the original data type (e.g. INT, YEAR) without
conversion to BIGINT.
This change is valid, because a number of recent fixes
(e.g. in Item_func_int, Item_hybrid_func, Item_int, Item_splocal)
guarantee that item->type_handler() now properly returns
type_handler_long vs type_handler_longlong. So no adjustment by length
is needed any more for Items returning INT_RESULT.
After this change, Item::create_tmp_field() calls
tmp_table_field_from_field_type() for all XXX_RESULT, except REAL_RESULT.
2. Fixing Item::create_tmp_field() not to have a special case for REAL_RESULT.
Note, the reason for a special case for REAL_RESULT is to have a special
constructor for Field_double(), forcing Field_real::not_fixed to be set
to true.
Taking into account that only Item_sum descendants actually need a special
constructor call Field_double(not_fixed=true), not too loose precision
when mixing individual rows to the aggregate result:
- renaming Item::create_tmp_field() to Item_sum::create_tmp_field().
- changing Item::create_tmp_field() just to call
tmp_table_field_from_field_type() for all XXX_RESULT types.
A special case for REAL_RESULT in Item::create_tmp_field() is now gone.
Item::create_tmp_field() is now symmetric for all XXX_RESULT types,
and now just calls tmp_table_field_from_field_type().
3. Fixing Item_func::create_field_for_create_select() not to have
a special case for STRING_RESULT.
After changes #1 and #2, the code in
Item_func::create_field_for_create_select(), testing result_type(),
becomes useless, because: now Item::create_tmp_field() and
tmp_table_field_from_field_type() do exactly the same thing for all
XXX_RESULT types for Item_func descendants:
a. It calls tmp_table_field_from_field_type for STRING_RESULT directly.
b. For other XXX_RESULT, it goes through Item::create_tmp_field(),
which calls the global function ::create_tmp_field(),
which calls item->create_tmp_field() for FUNC_ITEM,
which calls tmp_table_field_from_field_type() again.
So removing the virtual implementation of
Item_func::create_field_for_create_select().
The inherited Item::create_field_for_create_select() now perfectly
does the job, as it also calls tmp_table_field_from_field_type()
for FUNC_ITEM, independently from XXX_RESULT type.
4. Taking into account #1 and #2, as well as some recent changes,
removing virtual implementations:
- Item_hybrid_func::create_tmp_field()
- Item_hybrid_func::create_field_for_create_select()
- Item_int_func::create_tmp_field()
- Item_int_func::create_field_for_create_select()
- Item_temporal_func::create_field_for_create_select()
The derived versions from Item now perfectly work.
5. Moving a piece of code from create_tmp_field_from_item()
to a new function create_tmp_field_from_item_finalize(),
to reuse it in two places (see #6).
6. Changing the code responsible for BIT->INT/BIGIN tranformation
(which is called for the cases when the created table, e.g. HEAP,
does not fully support BIT) not to call create_tmp_field_from_item(),
because the latter now calls tmp_table_field_from_field_type() instead
of create_tmp_field() and thefore cannot do BIT transformation.
So rewriting this code using a sequence of these calls:
- item->type_handler_long_or_longlong()
- handler->make_and_init_table_field()
- create_tmp_field_from_item_finalize()
7. Miscelaneous changes:
- Moving type_handler_long_or_longlong() from "protected" to "public",
as it's now needed in the global function create_tmp_field().
8. The above changes fixed MDEV-12875, MDEV-12886, MDEV-12916.
So adding tests for these bugs.
MDEV-12858 Out-of-range error for CREATE..SELECT unsigned_int_column+1
MDEV-12859 Out-of-range error for CREATE..SELECT @a:=EXTRACT(MINUTE_MICROSECOND FROM..)
MDEV-12862 Data type of @a:=1e0 depends on the session character set
1. Moving a part of Item::create_tmp_field() into a new helper method
Item::create_tmp_field_int() and reusing it in Item::create_tmp_field()
and Item_func_signed::create_tmp_field().
Fixing the code in Item::create_tmp_field_int() to call
Type_handler::make_table_field() instead of doing "new Field_long[long]"
directly. This change revealed a problem reported in MDEV-12862.
2. Changing the "long vs longlong" cut-off length for
- Item_func::create_tmp_field()
- Item_sum::create_tmp_field()
- Item_func_get_user_var::create_tmp_field()
from MY_INT32_NUM_DECIMAL_DIGITS to (MY_INT32_NUM_DECIMAL_DIGITS - 2).
This fixes MDEV-12858.
After this change, the "convert_int_length" parameter to
Item::create_tmp_field() is not needed any more, because
(MY_INT32_NUM_DECIMAL_DIGITS - 2) is always passed.
So removing the "convert_int_length" parameter.
3. Fixing Item::create_tmp_field() to pass max_char_length() instead
of max_length to the constructor of Field_double().
This fixes MDEV-12862.
4. Additionally, fixing
- Type_handler_{tiny|short|int24|long|longlong}::make_table_field()
- Type_handler_{float|double}::make_table_field()
to pass max_char_length() instead of max_length to Field contructors.
This is needed by the change (1).
5. Adding new tests, and recording new correct results in the old tests in:
- mysql-test/r/type_ranges.result
- storage/tokudb/mysql-test/tokudb/r/type_ranges.result
Significantly reduce the amount of InnoDB, XtraDB and Mariabackup
code changes by defining pfs_os_file_t as something that is
transparently compatible with os_file_t.
At some conditions the function opt_sum_query() can apply MIN/MAX
optimizations to to Item_sum objects of a select These optimizations
becomes invalid if this select is the subquery of an IN subquery
predicate that is converted to a EXISTS subquery. Thus in this case
the MIX/MAX optimizations that have been applied in opt_sum_query()
must be rolled back.
This bug appeared in 5.3 when the code for the cost base choice between
materialization and in-to-exists transformation of non-correlated
IN subqueries was introduced. Before this code in-to-exists
transformations were always performed before the call of opt_sum_query().
1. Implementing the task according to the description:
a. Adding Type_handler::type_handler_for_tmp_table().
b. Adding Type_handler::type_handler_for_union_table.
c. Adding helper methods Type_handler::varstring_type_handler(const Item*),
Type_handler::blob_type_handler(const Item*)
d. Removing Item::make_string_field() and
Item_func_group_concat::make_string_field().
They are not needed any more.
e. Simplifying Item::tmp_table_field_from_field_type() to just two lines.
f. Renaming Item_type_holder::make_field_by_type() and implementing
virtual Item_type_holder::create_tmp_field() instead.
The new implementation is also as simple as two lines.
g. Adding a new virtual method Type_all_attributes::get_typelib(),
to access to TYPELIB definitions for ENUM and SET columns.
h. Simplifying the code branch for TIME_RESULT, DECIMAL_RESULT, STRING_RESULT
in Item::create_tmp_field(). It's now just one line.
i. Implementing Type_handler_enum::make_table_field() and
Type_handler_set::make_table_field().
2. Code simplification in Field_str constructor calls.
a. Changing the "CHARSET_INFO *cs" argument in constuctors for Field_str
and its descendants to "const DTCollation &collation". This is to
avoid two step initialization:
- setting Field_str::derivation and Field_str::repertoire to the
default values first
- then resetting them using:
set_derivation(item->derivation, item->repertoire).
b. Removing Field::set_derivation()
c. Adding a new constructor DTCollation(CHARSET_INFO *cs),
for the old code compatibility.
3. Changes in test results
As a side effect some test results have changed, because
in the old version Item::make_string_field() converted
TINYBLOB to VARCHAR(255). Now TINYBLOB is preserved.
a. sp-row.result
This query:
CREATE TABLE t1 AS SELECT tinyblob_sp_variable;
Now preserves TINYBLOB as the data type.
Before the patch a VARCHAR(255) was created.
b. gis-debug.result
This is a debug test, to make sure that + and - operators
are commutative and non-commutative correspondingly.
The exact data type is not really important.
(But anyway, it now chooses a better data type that fits the result)
Benefits of this patch:
- Removed a lot of calls to strlen(), especially for field_string
- Strings generated by parser are now const strings, less chance of
accidently changing a string
- Removed a lot of calls with LEX_STRING as parameter (changed to pointer)
- More uniform code
- Item::name_length was not kept up to date. Now fixed
- Several bugs found and fixed (Access to null pointers,
access of freed memory, wrong arguments to printf like functions)
- Removed a lot of casts from (const char*) to (char*)
Changes:
- This caused some ABI changes
- lex_string_set now uses LEX_CSTRING
- Some fucntions are now taking const char* instead of char*
- Create_field::change and after changed to LEX_CSTRING
- handler::connect_string, comment and engine_name() changed to LEX_CSTRING
- Checked printf() related calls to find bugs. Found and fixed several
errors in old code.
- A lot of changes from LEX_STRING to LEX_CSTRING, especially related to
parsing and events.
- Some changes from LEX_STRING and LEX_STRING & to LEX_CSTRING*
- Some changes for char* to const char*
- Added printf argument checking for my_snprintf()
- Introduced null_clex_str, star_clex_string, temp_lex_str to simplify
code
- Added item_empty_name and item_used_name to be able to distingush between
items that was given an empty name and items that was not given a name
This is used in sql_yacc.yy to know when to give an item a name.
- select table_name."*' is not anymore same as table_name.*
- removed not used function Item::rename()
- Added comparision of item->name_length before some calls to
my_strcasecmp() to speed up comparison
- Moved Item_sp_variable::make_field() from item.h to item.cc
- Some minimal code changes to avoid copying to const char *
- Fixed wrong error message in wsrep_mysql_parse()
- Fixed wrong code in find_field_in_natural_join() where real_item() was
set when it shouldn't
- ER_ERROR_ON_RENAME was used with extra arguments.
- Removed some (wrong) ER_OUTOFMEMORY, as alloc_root will already
give the error.
TODO:
- Check possible unsafe casts in plugin/auth_examples/qa_auth_interface.c
- Change code to not modify LEX_CSTRING for database name
(as part of lower_case_table_names)
Due to this bug many queries that contained a window function
with MIN/MAX aggregation returned wrong results.
Calculation of a MIN/MAX aggregate function uses cache objects
and a comparator object that are created and set up in
Item_sum_hybrid::fix_fields () by a call of Item_sum_hybrid::setup_hybrid().
The latter binds the objects to the first argument of the
MIN/MAX function. Meanwhile window function perform aggregation
over fields of a temporary table. So binding must be done rather to
these fields. The earliest moment when setup the objects used in
MIN/max functions can be done is after all calls of the method
split_sum_func().
This patch introduces this late setup, but only for aggregate
functions used in window functions.
Probably it makes sense to use this late setup for all MIN/MAX
objects.
