At the second execution of the PS
1. mark_as_dependent() is called with the same parameters as at the first
execution (select#4 and select#3)
2. as outer_select (select#3) has been already merged at the first
execution of PS it cannot be reached using the outer_select() function
anymore (and so can not stop iteration).
3. as a result all selects towards the top level select including the
select for 'ca' are marked as uncacheable.
4. Marked uncacheable it executed incorrectly triggering filling its
temporary table several times and using freed memory at the end.
To avoid the problem we use name resolution context to go "up".
NOTE: problem also exists in 10.2 but has no visible effect on execution.
That is why the problem is fixed in 10.2.
The patch also add debug logging of important procedures and
better specify parameters types of st_select_lex::mark_as_dependent.
The EXPLAIN EXTENDED statement run as a prepared statement can produce extra
warning comparing with a case when EXPLAIN EXTENDED statement is run as
a regular statement. For example, the following test case
CREATE TABLE t1 (c int);
CREATE TABLE t2 (d int);
EXPLAIN EXTENDED SELECT (SELECT 1 FROM t2 WHERE d = c) FROM t1;
produces the extra warning
"Field or reference 'c' of SELECT #2 was resolved in SELECT #1"
in case the above mentioned "EXPLAIN EXTENDED" statement is executed
in PS mode, that is by submitting the following statements:
PREPARE stmt FROM "EXPLAIN EXTENDED SELECT (SELECT 1 FROM t2 WHERE d = c) FROM t1";
EXECUTE stmt;
The reason of the extra warning emittion is in a way items
are handled (being fixed) during execution of the JOIN::prepare() method.
The method Item_field::fix_fields() calls the find_field_in_tables()
function in case a field hasn't been associated yet with the item.
Implementation of the find_field_in_tables() function first checks whether
a table containing the required field was already opened and cached.
It is done by checking the data member item->cached_table. This data member
is set on handling the PRERARE FROM statement and checked on executing
the EXECUTE statement. If the data member item->cached_table is set
the find_field_in_tables() function invoked and the
mark_select_range_as_dependent() function called if the field
is an outer referencee. The mark_select_range_as_dependent() function
calls the mark_as_dependent() function that finally invokes
the push_warning_printf() function that produces extra warning.
To fix the issue, calling of push_warning_printf() is elimited in case
it was run indirectly in result of hanlding already opened table from
the Item_field::fix_fields() method.
The query causing the issue here has implicit grouping for we
have to produce one row with special values for the aggregates
(depending on each aggregate function), and NULL values for all
non-aggregate fields.
The subselect item where implicit grouping was being done,
null_value for the subselect item was not being set for
the case when the implicit grouping produces NULL values
for the items in the select list of the subquery.
This which was leading to the crash.
The fix would be to set the null_value when all the values
for the row column have NULL values.
Further changes are
1) etting null_value for Item_singlerow_subselect only
after val_* functions have been called.
2) Introduced a parameter null_value_inside to Item_cache that
would store be set to TRUE if any of the arguments of the
Item_cache are null.
Reviewed And co-authored by Monty
Introduced val_time_packed and val_datetime_packed functions for Item_direct_ref
to make sure to get the value from the item it is referring to.
The issue for incorrect result was that the item was getting its value
from the temporary table rather than from the view.
The assertion failed in handler::ha_reset upon SELECT under
READ UNCOMMITTED from table with index on virtual column.
This was the debug-only failure, though the problem is mush wider:
* MY_BITMAP is a structure containing my_bitmap_map, the latter is a raw
bitmap.
* read_set, write_set and vcol_set of TABLE are the pointers to MY_BITMAP
* The rest of MY_BITMAPs are stored in TABLE and TABLE_SHARE
* The pointers to the stored MY_BITMAPs, like orig_read_set etc, and
sometimes all_set and tmp_set, are assigned to the pointers.
* Sometimes tmp_use_all_columns is used to substitute the raw bitmap
directly with all_set.bitmap
* Sometimes even bitmaps are directly modified, like in
TABLE::update_virtual_field(): bitmap_clear_all(&tmp_set) is called.
The last three bullets in the list, when used together (which is mostly
always) make the program flow cumbersome and impossible to follow,
notwithstanding the errors they cause, like this MDEV-17556, where tmp_set
pointer was assigned to read_set, write_set and vcol_set, then its bitmap
was substituted with all_set.bitmap by dbug_tmp_use_all_columns() call,
and then bitmap_clear_all(&tmp_set) was applied to all this.
To untangle this knot, the rule should be applied:
* Never substitute bitmaps! This patch is about this.
orig_*, all_set bitmaps are never substituted already.
This patch changes the following function prototypes:
* tmp_use_all_columns, dbug_tmp_use_all_columns
to accept MY_BITMAP** and to return MY_BITMAP * instead of my_bitmap_map*
* tmp_restore_column_map, dbug_tmp_restore_column_maps to accept
MY_BITMAP* instead of my_bitmap_map*
These functions now will substitute read_set/write_set/vcol_set directly,
and won't touch underlying bitmaps.
This piece of the code in Item_func_or_sum::agg_item_set_converter:
if (!conv && ((*arg)->collation.repertoire == MY_REPERTOIRE_ASCII))
conv= new (thd->mem_root) Item_func_conv_charset(thd, *arg, coll.collation, 1);
was wrong because:
1. It could change Item_cache to Item_func_conv_charset
(with the old Item_cache in args[0]).
Such Item type change is not always supported, e.g.
the code in Item_singlerow_subselect::reset() expects only Item_cache,
to be able to call Item_cache::set_null(). So it erroneously
reinterpreted Item_func_conv_charset to Item_cache and called
a non-existing method set_null(), which crashed the server.
2. The 1 in the last parameter to Item_func_conv_charset() was also
a problem. In MariaDB versions where the reported query did not crash,
it erroneously returned "empty set" instead of one row, because
the 1 made subselects execute too earlier and return NULL.
Fix:
1. Removing the above two lines from Item_func_or_sum::agg_item_set_converter()
2. Adding the repertoire test inside the constructor of Item_func_conv_charset,
so it now detects itself as "safe" in more cases than before.
This is needed to avoid new "Illegal mix of collations" after
removing the wrong code in various scenarios when character set
conversion from pure ASCII happens, including the reported scenario.
So now this sequence:
Item_cache -> Item_func_concat
is replaced to this compatible sequence (the top Item is still Item_cache):
new Item_cache -> Item_func_conv_charset -> Item_func_concat
Before the fix it was replaced to this incompatible sequence:
Item_func_conv_charset -> old Item_cache -> Item_func_concat
* Allocate items on thd->mem_root while refixing vcol exprs
* Make vcol tree changes register and roll them back after the statement is executed.
Explanation:
Due to collation implementation specifics an Item tree could change while fixing.
The tricky thing here is to make it on a proper arena.
It's usually not a problem when a field is deterministic, however, makes a pain vice-versa, during allocation allocating.
A non-deterministic field should be refixed on each statement, since it depends on the environment state.
Changing the tree will be temporary and therefore it should be reverted after the statement execution.
prepared statement
The method Item_func_in::build_clone() that builds a clone item for an
Item_func_in item first calls a generic method Item_func::build_item()
that builds the the clones for the arguments of the Item_func_in item
to be cloned, creates a copy of the Item_func_in object and attaches the
clones for the arguments to this copy. Then the method Item_func_in::build_clone()
makes the copy fully independent on the copied object in order to
guarantee a proper destruction of the clone. The fact is the copy of the
Item_func_in object is registered as any other item object and should be
destructed as any other item object.
If the method Item_func::build_item fails to build a clone of an argument
then it returns 0. In this case no copy of the Item_func_in object should
be created. Otherwise the finalizing actions for this copy would not be
performed and the copy would remain in a state that would prevent its
proper destruction.
The code of Item_func_in::build_clone() before this patch created the copy
of the Item_func_in object before cloning the argument items. If this
cloning failed the server crashed when trying to destruct the copy item.
The code of Item_row::build_clone() was changed similarly to the code of
Item_func::build_clone though this code could not cause any problems.
Item_ref::val_(datetime|time)_packed() erroneously called
(*ref)->val_(datetime|time)_packed().
- Fixing to call (*ref)->val_(datetime|time)_packed_result().
- Backporting Item::val_(datetime|time)_packed_result() from 10.3.
- Fixing Item_field::get_date_result() to handle null_value in the same
way how Item_field::get_date() does.
The issue here is window function makes the passed string object
to point to an area in a temporary table's record buffer.
Then, the temporary table is freed, together with its record buffer.
Then, Item_cache_str attempts to read this value.
The fix is to call value_buff.copy(). This will make the value_buff to store
its string in a buffer that it owns, which will not disappear unexpectedly.
remove a special treatment of a bare DEFAULT keyword that made it
behave inconsistently and differently from DEFAULT(column).
Now all forms of the explicit assignment of a default column value
behave identically, and all count as an explicitly assigned value
(for the purpose of ON UPDATE NOW).
followup for c7c481f4d9
and WHERE filter afterwards
This patch complements the patch fixing the bug MDEV-6892. The latter
properly handled queries that used mergeable views returning constant
columns as inner tables of outer joins and whose where clause contained
predicates referring to these columns if the predicates of happened not
to be equality predicates. Otherwise the server still could return wrong
result sets for such queries. Besides the fix for MDEV-6892 prevented
some possible conversions of outer joins to inner joins for such queries.
This patch corrected the function check_simple_equality() to handle
properly conjunctive equalities of the where clause that refer to the
constant columns of mergeable views used as inner tables of an outer join.
The patch also changed the code of Item_direct_view_ref::not_null_tables().
This change allowed to take into account predicates containing references
to constant columns of mergeable views when converting outer joins into
inner joins.
The issue here was when we had a subquery and a window function in an expression in
the select list then subquery was getting computed after window function computation.
This resulted in incorrect results because the subquery was correlated and the fields
in the subquery was pointing to the base table instead of the temporary table.
The approach to fix this was to have an additional field in the temporary table
for the subquery and to execute the subquery before window function execution.
After execution the values for the subquery were stored in the temporary table
and then when we needed to calcuate the expression, all we do is read the values
from the temporary table for the subquery.
ASAN noticed a freed memory access during EXECUTE in this script:
PREPARE stmt FROM "SELECT 'x' ORDER BY NAME_CONST( 'f', 'foo' )";
EXECUTE stmt;
In case of a PREPARE statement, all Items, including Item_name_const,
are created on Prepared_statement::main_mem_root.
Item_name_const::fix_fields() did not take this into account
and could allocate the value of Item::name on a wrong memory root,
in this code:
if (is_autogenerated_name)
{
set_name(thd, item_name->c_ptr(), (uint) item_name->length(),
system_charset_info);
}
When fix_fields() is called in the reported SQL script, THD's arena already
points to THD::main_mem_root rather than to Prepared_statement::main_mem_root,
so Item::name was allocated on THD::main_mem_root.
Then, at the end of the dispatch_command() for the PREPARE statement,
THD::main_mem_root got cleared. So during EXECUTE, Item::name
pointed to an already freed memory.
This patch changes the code to set the implicit name for Item_name_const
at the constructor time rather than at fix_fields time. This guarantees
that Item_name_const and its Item::name always reside on the same memory root.
Note, this change makes the code for Item_name_const symmetric with other
constant-alike items that set their default implicit names at the constructor
call time rather than at fix_fields() time:
- Item_string
- Item_int
- Item_real
- Item_decimal
- Item_null
- Item_param
While printing a view containing a window function we were printing it as an
Item_field object instead of an Item_window_func object. This is incorrect and this
leads to us throwing an error ER_VIEW_INVALID.
Fixed by adjusting the Item_ref:print function.
Also made UDF function aware if there arguments have window function.
The error message modified.
Then the TABLE_SHARE::error_table_name() implementation taken from 10.3,
to be used as a name of the table in this message.
- clean up DEFAULT() to work only with default value and correctly print
itself.
- fix of DBUG_ASSERT about fields read/write
- fix of field marking for write based really on the thd->mark_used_columns flag
Item_direct_view_ref::derived_field_transformer_for_where
upon updating a view
The condition pushed into a materialized derived / view mast be adjusted
for the new context: its column references must be substituted for
references to the columns of the underlying tables if the condition
is pushed into WHERE. The substitution is performed by the 'transform'
method. If the materialized derived is used in a mergeable view then
the references to the columns of the view are represented by
Item_direct_view_ref objects. The transform method first processes
the item wrapped in such an object and only after this it transforms
the object itself.
The transformation procedure of an Item_direct_view_ref object has
to know whether the item it wraps has been substituted. If so the
procedure does not have to do anything. In the code before this patch
it was not possible for the transformation procedure used by an
Item_direct_view_ref object to find out whether a substitution for
the wrapped item had happened.
upon INSERT .. SELECT
The function Item *Item_direct_view_ref::derived_field_transformer_for_where()
erroneously did not strip off ref wrappers from references to materialized
derived tables / views. As a result the expressions that contained some
references of the type Item_direct_view_ref to columns of a materialized
derived table / view V were pushed into V incorrectly. This could cause
crashes for some INSERT ... SELECT statements.
This patch fills a serious flaw in the implementation of common table
expressions. Before this patch an attempt to prepare a statement from
a query with a parameter marker in a CTE that was used more than once
in the query ended up with a bogus error message. Similarly if a statement
in a stored procedure contained a CTE whose specification used a
local variables and this CTE was referred to more than once in the
statement then the server failed to execute the stored procedure returning
a bogus error message on a non-existing field.
The problems appeared due to incorrect handling of parameter markers /
local variables in CTEs that were referred more than once.
This patch fixes the problems by differentiating between the original
occurrences of a parameter marker / local variable used in the
specification of a CTE and the corresponding occurrences used
in copies of this specification. These copies are substituted
instead of non-first references to the CTE.
The idea of the fix and even some code were taken from the MySQL
implementation of the common table expressions.
