Synopsis: If SELECT returned answer from Query Cache it is not really executed.
The reason for firing of assertion
DBUG_ASSERT((mem_root->flags & ROOT_FLAG_READ_ONLY) == 0);
is that in case the query_cache is on and the same query run by different
stored routines the following use case can take place:
First, lets say that bodies of routines used by the test case are the same
and contains the only query 'SELECT * FROM t1';
call p1() -- a result set is stored in query cache for further use.
call p2() -- the same query is run against the table t1, that result in
not running the actual query but using its cached result.
On finishing execution of this routine, its memory root is
marked for read only since every SP instruction that this
routine contains has been executed.
INSERT INT t1 VALUE (1); -- force following invalidation of query cache
call p2() -- query the table t1 will result in assertion failure since its
execution would require allocation on the memory root that
has been already marked as read only memory root
The root cause of firing the assertion is that memory root of the stored
routine 'p2' was marked as read only although actual execution of the query
contained inside hadn't been performed.
To fix the issue, mark a SP instruction as not yet run in case its execution
doesn't result in real query processing and a result set got from query cache
instead.
Note that, this issue relates server built in debug mode AND with the protect
statement memory root feature turned on. It doesn't affect server built
in release mode.
This patch adds support for controlling of memory allocation
done by SP/PS that could happen on second and following executions.
As soon as SP or PS has been executed the first time its memory root
is marked as read only since no further memory allocation should
be performed on it. In case such allocation takes place it leads to
the assert hit for invariant that force no new memory allocations
takes place as soon as the SP/PS has been marked as read only.
The feature for control of memory allocation made on behalf SP/PS
is turned on when both debug build is on and the cmake option
-DWITH_PROTECT_STATEMENT_MEMROOT is set.
The reason for introduction of the new cmake option
-DWITH_PROTECT_STATEMENT_MEMROOT
to control memory allocation of second and following executions of
SP/PS is that for the current server implementation there are too many
places where such memory allocation takes place. As soon as all such
incorrect allocations be fixed the cmake option
-DWITH_PROTECT_STATEMENT_MEMROOT
can be removed and control of memory allocation made on second and
following executions can be turned on only for debug build. Before
every incorrect memory allocation be fixed it makes sense to guard
the checking of memory allocation on read only memory by extra cmake
option else we would get a lot of failing test on buildbot.
Moreover, fixing of all incorrect memory allocations could take pretty
long period of time, so for introducing the feature without necessary
to wait until all places throughout the source code be fixed it makes
sense to add the new cmake option.
- Moving the code from a public function trim_whitespaces()
to the class Lex_cstring as methods. This code may
be useful in other contexts, and also this code becomes
visible inside sql_class.h
- Adding a helper method THD::strmake_lex_cstring_trim_whitespaces()
- Unifying the way how CREATE PROCEDURE/CREATE FUNCTION and
CREATE PACKAGE/CREATE PACKAGE BODY work:
a) Now CREATE PACKAGE/CREATE PACKAGE BODY also calls
Lex->sphead->set_body_start() to remember the cpp body start inside
an sp_head member.
b) adding a "const char *cpp_body_end" parameter to
sp_head::set_stmt_end().
These changes made it possible to reuse sp_head::set_stmt_end() inside
LEX::create_package_finalize() and remove the duplucate code.
- Renaming sp_head::m_body_begin to m_cpp_body_begin and adding a comment
to make it clear that this member is used only during parsing, and
points to a fragment inside the cpp buffer.
- Changed sp_head::set_body_start() and sp_head::set_stmt_end()
to skip the calls related to "body_utf8" in cases when m_parent is not NULL.
A non-NULL m_parent means that we're inside a package routine.
"body_utf8" in such case belongs not to the current sphead itself,
but to parent (the package) sphead.
So an sphead instance of a package routine should neither initialize,
nor finalize, nor change in any other ways the "body_utf8" related
members of Lex_input_stream, and should not take over or copy "body_utf8"
data from Lex_input_stream to "this".
The parser works as follows:
The rule expr_lex returns a pointer to a newly created sp_expr_lex
instance which is not linked to any MariaDB structures yet - it is
pointed only from a Bison stack variable. The sp_expr_lex instance
gets linked to other structures (such as sp_instr_jump_if_not) later,
after scanning some following grammar.
Problem before the fix:
If a parse error happened immediately after expr_lex (before it got linked),
the created sp_expr_lex value got lost causing a memory leak.
Fix:
- Using Bison's "destructor" directive to free the results of expr_lex
on parse/oom errors.
- Moving the call for LEX::cleanup_lex_after_parse_error() from
MYSQL_YYABORT and yyerror inside parse_sql().
This is needed because Bison calls destructors after yyerror(),
while it's important to delete the sp_expr_lex instance before
LEX::cleanup_lex_after_parse_error().
The latter frees the memory root containing the sp_expr_lex instance.
After this change the code block are executed in the following order:
- yyerror() -- now only raises the error to DA (no cleanup done any more)
- %destructor { delete $$; } <expr_lex> -- destructs the sp_expr_lex instance
- LEX::cleanup_lex_after_parse_error() -- frees the memory root containing
the sp_expr_lex instance
- Removing the "delete sublex" related code from restore_lex():
- restore_lex() is called in most cases on success, when delete is not needed.
- There is one place when restore_lex() is called on error:
In sp_create_assignment_instr(). But in this case LEX::sp_lex_in_use
is true anyway.
The patch adds a new DBUG_ASSERT(lex->sp_lex_in_use) to guard this.
This patch is the result of running
run-clang-tidy -fix -header-filter=.* -checks='-*,modernize-use-equals-default' .
Code style changes have been done on top. The result of this change
leads to the following improvements:
1. Binary size reduction.
* For a -DBUILD_CONFIG=mysql_release build, the binary size is reduced by
~400kb.
* A raw -DCMAKE_BUILD_TYPE=Release reduces the binary size by ~1.4kb.
2. Compiler can better understand the intent of the code, thus it leads
to more optimization possibilities. Additionally it enabled detecting
unused variables that had an empty default constructor but not marked
so explicitly.
Particular change required following this patch in sql/opt_range.cc
result_keys, an unused template class Bitmap now correctly issues
unused variable warnings.
Setting Bitmap template class constructor to default allows the compiler
to identify that there are no side-effects when instantiating the class.
Previously the compiler could not issue the warning as it assumed Bitmap
class (being a template) would not be performing a NO-OP for its default
constructor. This prevented the "unused variable warning".
Problem:
At some point, we made stored rountines fail at CREATE time
instead of execution time in case of this syntax:
IF unknown_variable
...
END IF
As a result, a trigger created before this change and contained an unknown
variable worked in a bad way after upgrade:
- It was displayed with an empty trigger name by SHOW CREATE TRIGGER
- It was displayed with an empty trigger name by INFORMATION_SCHEMA.TRIGGERS
- An attempt to DROP this trigger returned errors - nothing happened.
- DROP TABLE did not remove the .TRN file corresponding to this broken trigger.
Underlying code observations:
The old code assumed that the trigger name resides in the current lex:
if(thd->lex->spname)
m_trigger_name= &thd->lex->spname->m_name;
This is not always the case. Some SP statements (e.g. IF)
do the following in their beginning:
- create a separate local LEX
- set thd->lex to this new local LEX
- push the new local LEX to the stack in sp_head::m_lex
and the following at the end of the statement:
- pop the previous LEX from the stack sp_head::m_lex
- set thd->lex back to the popped value
So when the parse error happens inside e.g. IF statement, thd->lex->spname
is a NULL pointer, because thd->lex points to the local LEX (without SP name)
rather than the top level LEX (with SP name).
Fix:
- Adding a new method sp_head::find_spname_recursive()
which walks inside the LEX stack sp_head::m_lex from
the top (the newest, most local) to the bottom (the oldest),
and finds the one which contains a non-zero spname pointer.
- Using the new method inside
Deprecated_trigger_syntax_handler::handle_condition():
First it still tests thd->lex->spname (like before this change),
and uses it in case it is not empty.
Otherwise (if thd->lex->spname is empty), it calls
sp_head::find_spname_recursive() to find the LEX with a
non-empty spname inside the LEX stack of the current sphead.
(Variant #2 of the patch, which keeps the sp_head object inside the
MEM_ROOT that sp_head object owns)
(10.3 requires extra work due to sp_package, will commit a separate
patch for it)
sp_head::operator new() and operator delete() were dereferencing sp_head*
pointers to memory that didn't hold a valid sp_head object (it was
not created/already destroyed).
This caused UBSan to crash when looking up type information.
Fixed by providing static sp_head::create() and sp_head::destroy() methods.
(Variant #2 of the patch, which keeps the sp_head object inside the
MEM_ROOT that sp_head object owns)
(10.3 version of the fix, with handling for class sp_package)
sp_head::operator new() and operator delete() were dereferencing sp_head*
pointers to memory that didn't hold a valid sp_head object (it was
not created/already destroyed).
This caused UBSan to crash when looking up type information.
Fixed by providing static sp_head::create() and sp_head::destroy() methods.
Part#2 (final): rewritting the code to pass the correct enum_sp_aggregate_type
to the sp_head constructor, so sp_head never changes its aggregation type
later on. The grammar has been simplified and defragmented.
This allowed to check aggregate specific instructions right after
a routine body has been scanned, by calling new LEX methods:
sp_body_finalize_{procedure|function|trigger|event}()
Moving some C++ code from *.yy to a few new helper methods in LEX.
This task involves the implementation for the optimizer trace.
This feature produces a trace for any SELECT/UPDATE/DELETE/,
which contains information about decisions taken by the optimizer during
the optimization phase (choice of table access method, various costs,
transformations, etc). This feature would help to tell why some decisions were
taken by the optimizer and why some were rejected.
Trace is session-local, controlled by the @@optimizer_trace variable.
To enable optimizer trace we need to write:
set @@optimizer_trace variable= 'enabled=on';
To display the trace one can run:
SELECT trace FROM INFORMATION_SCHEMA.OPTIMIZER_TRACE;
This task also involves:
MDEV-18489: Limit the memory used by the optimizer trace
introduces a switch optimizer_trace_max_mem_size which limits
the memory used by the optimizer trace. This was implemented by
Sergei Petrunia.
Problem:
push_handler() created sp_handler_entry instances on THD::main_mem_root,
which is freed only after the SP instructions execution.
So in case of a CONTINUE HANDLER inside a loop (e.g. WHILE) this approach
leaked thread memory on every loop iteration.
Changes:
- Removing sp_handler_entry declaration, it's not really needed.
- Fixing the data type of sp_rcontext::m_handlers from
Dynamic_array<sp_handler_entry*> to Dynamic_array<sp_instr_hpush_jump*>
- Fixing sp_rcontext::push_handler() to push the pointer to
an sp_instr_hpush_jump instance to the handler stack.
This instance contains everything we need.
There is no a need to allocate anything else.
Problem:
push_cursor() created sp_cursor instances on THD::main_mem_root,
which is freed only after the SP instructions loop.
Changes:
- Moving sp_cursor declaration from sp_rcontext.h to sql_class.h
- Deriving sp_instr_cpush from sp_cursor. So now sp_cursor is created
only once (at the SP parse time) and then reused on all loop iterations
- Adding a new method reset() into sp_cursor (and its parent classes)
to reset an sp_cursor instance before reuse.
- Moving former sp_cursor members m_fetch_count, m_row_count, m_found
into a separate class sp_cursor_statistics. This helps to reuse
the code in sp_cursor constructors, and in sp_cursor::reset()
- Adding a helper method sp_rcontext::pop_cursor().
- Adding "THD*" parameter to so_rcontext::pop_cursors() and pop_all_cursors()
- Removing "new" and "delete" from sp_rcontext::push_cursor() and
sp_rconext::pop_cursor().
- Fixing sp_cursor not to derive from Sql_alloc, as it's now allocated
only as a part of sp_instr_cpush (and not allocated separately).
- Moving lex_keeper->disable_query_cache() from sp_cursor::sp_cursor()
to sp_instr_cpush::execute().
- Adding tests
MDEV-10581 sql_mode=ORACLE: Explicit cursor FOR LOOP
MDEV-12098 sql_mode=ORACLE: Implicit cursor FOR loop
Cleanup changes:
- Removing sp_lex_cursor::m_cursor_name
- Adding sp_instr_cursor_copy_struct::m_cursor (the cursor global index)
- Fixing sp_instr_cursor_copy_struct::print() to access to the cursor
name using m_ctx and m_cursor (like other cursor related instructions do)
instead of m_cursor_name.
This change is needed to unify sp_assignment_lex and sp_cursor_lex later,
to fix this problem easier:
MDEV-16558 Parenthesized expression does not work as a lower FOR loop bound
The print() function was missing from the FETCH GROUP NEXT ROW instrunction class, so there was no
output for this particular instruction when we use the query SHOW FUNCTION CODE function_name
- CREATE PACKAGE [BODY] statements are now
entirely written to mysql.proc with type='PACKAGE' and type='PACKAGE BODY'.
- CREATE PACKAGE BODY now supports IF NOT EXISTS
- DROP PACKAGE BODY now supports IF EXISTS
- CREATE OR REPLACE PACKAGE [BODY] is now supported
- CREATE PACKAGE [BODY] now support the DEFINER clause:
CREATE DEFINER user@host PACKAGE pkg ... END;
CREATE DEFINER user@host PACKAGE BODY pkg ... END;
- CREATE PACKAGE [BODY] now supports SQL SECURITY and COMMENT clauses, e.g.:
CREATE PACKAGE p1 SQL SECURITY INVOKER COMMENT "comment" AS ... END;
- Package routines are now created from the package CREATE PACKAGE BODY
statement and don't produce individual records in mysql.proc.
- CREATE PACKAGE BODY now supports package-wide variables.
Package variables can be read and set inside package routines.
Package variables are stored in a separate sp_rcontext,
which is cached in THD on the first packate routine call.
- CREATE PACKAGE BODY now supports the initialization section.
- All public routines (i.e. declared in CREATE PACKAGE)
must have implementations in CREATE PACKAGE BODY
- Only public package routines are available outside of the package
- {CREATE|DROP} PACKAGE [BODY] now respects CREATE ROUTINE and ALTER ROUTINE
privileges
- "GRANT EXECUTE ON PACKAGE BODY pkg" is now supported
- SHOW CREATE PACKAGE [BODY] is now supported
- SHOW PACKAGE [BODY] STATUS is now supported
- CREATE and DROP for PACKAGE [BODY] now works for non-current databases
- mysqldump now supports packages
- "SHOW {PROCEDURE|FUNCTION) CODE pkg.routine" now works for package routines
- "SHOW PACKAGE BODY CODE pkg" now works (the package initialization section)
- A new package body level MDL was added
- Recursive calls for package procedures are now possible
- Routine forward declarations in CREATE PACKATE BODY are now supported.
- Package body variables now work as SP OUT parameters
- Package body variables now work as SELECT INTO targets
- Package body variables now support ROW, %ROWTYPE, %TYPE
Handle string length as size_t, consistently (almost always:))
Change function prototypes to accept size_t, where in the past
ulong or uint were used. change local/member variables to size_t
when appropriate.
This fix excludes rocksdb, spider,spider, sphinx and connect for now.
Counter for select numbering made stored with the statement (before was global)
So now it does have always accurate value which does not depend on
interruption of statement prepare by errors like lack of table in
a view definition.
This was done in, among other things:
- thd->db and thd->db_length
- TABLE_LIST tablename, db, alias and schema_name
- Audit plugin database name
- lex->db
- All db and table names in Alter_table_ctx
- st_select_lex db
Other things:
- Changed a lot of functions to take const LEX_CSTRING* as argument
for db, table_name and alias. See init_one_table() as an example.
- Changed some function arguments from LEX_CSTRING to const LEX_CSTRING
- Changed some lists from LEX_STRING to LEX_CSTRING
- threads_mysql.result changed because process list_db wasn't always
correctly updated
- New append_identifier() function that takes LEX_CSTRING* as arguments
- Added new element tmp_buff to Alter_table_ctx to separate temp name
handling from temporary space
- Ensure we store the length after my_casedn_str() of table/db names
- Removed not used version of rename_table_in_stat_tables()
- Changed Natural_join_column::table_name and db_name() to never return
NULL (used for print)
- thd->get_db() now returns db as a printable string (thd->db.str or "")
After MDEV-14212, the Virtual_tmp_table instance that stores a ROW
variable elements is accessible from the underlying Field_row
(rather than Item_field_row).
This patch makes some further changes by moving the code from
sp_instr_xxx, sp_rcontext, Item_xxx to Virtual_tmp_table and Field_xxx.
The data type specific code (scalar vs ROW) now resides in
a new virtual method Field_xxx::sp_prepare_and_store_item().
The the code in sp_rcontext::set_variable() and sp_eval_expr()
is now symmetric for scalar and ROW values.
The code in sp_rcontext::set_variable_row_field(), sp_rcontext::set_variable_row_field(), sp_rcontext::set_variable_row()
is now symmetric for ROW elements (i.e. scalar and ROW elements inside a ROW).
Rationale:
Prepare the code to implement these tasks soon easier:
- MDEV-12252 ROW data type for stored function return values
- MDEV-12307 ROW data type for built-in function return values
- MDEV-6121 Data type: Array
- MDEV-10593 sql_mode=ORACLE: TYPE .. AS OBJECT: basic functionality
- ROW with ROW fields (no MDEV yet)
Details:
1. Moving the code in sp_eval_expr() responsible to backup/restore
thd->count_cuted_fields, thd->abort_on_warning,
thd->transaction.stmt.modified_non_trans_table
into a new helper class Sp_eval_expr_state, to reuse it easier.
Fixing sp_eval_expr() to use this new class.
2. Moving sp_eval_expr() and sp_prepare_func_item() from public functions
to methods in THD, so they can be reused in *.cc files easier without
a need to include "sp_head.h".
Splitting sp_prepare_func_item() into two parts.
Adding a new function sp_fix_func_item(), which fixes
the underlying items, but does not do check_cols() for them.
Reusing sp_fix_func_item() in Field_row::sp_prepare_and_store_item().
3. Moving the code to find ROW fields by name from Item to Virtual_tmp_table
Moving the code searching for ROW fields by their names
from Item_field_row::element_index_by_name() to a new method
Item_field_row to Virtual_tmp_table::sp_find_field_by_name().
Adding wrapper methods sp_rcontext::find_row_field_by_name() and
find_row_field_by_name_or_error(), to search for a ROW variable
fields by the variable offset and its field name.
Changing Item_splocal_row_field_by_name::fix_fields() to do
use sp_rcontext::find_row_field_by_name_or_error().
Removing virtual Item::element_index_by_name().
4. Splitting sp_rcontext::set_variable()
Adding a new virtual method Field::sp_prepare_and_store_item().
Spliting the two branches of the code in sp_rcontext::set_variable()
into two virtual implementations of Field::sp_prepare_and_store_item(),
(for Field and for Field_row).
Moving the former part of sp_rcontext::set_variable() with the loop
doing set_null() for all ROW fields into a new method
Virtual_tmp_table::set_all_fields_to_null() and using it in
Field_row::sp_prepare_and_store_item().
Moving the former part of sp_rcontext::set_variable() with the loop
doing set_variable_row_field() into a new method
Virtual_tmp_table::set_all_fields_from_item() and using it in
Field_row::sp_prepare_and_store_item().
The loop in the new method now uses sp_prepare_and_store_item()
instead of set_variable_row_field(), because saving/restoring
THD flags is now done on the upper level. No needs to save/restore
on every iteration.
5. Fixing sp_eval_expr() to simply do two things:
- backup/restore THD flags
- call result_field->sp_prepare_and_store_item()
So now sp_eval_expr() can be used for both scalar and ROW variables.
Reusing it in sp_rcontext::set_variable*().
6. Moving the loop in sp_rcontext::set_variable_row() into a
new method Virtual_tmp_table::sp_set_all_fields_from_item_list().
Changing the loop body to call field->sp_prepare_and_store_item()
instead of doing set_variable_row_field(). This removes
saving/restoring of the THD flags from every interation.
Instead, adding the code to save/restore the flags around
the entire loop in set_variable_row(), using Sp_eval_expr_state.
So now saving/restoring is done only once for the entire ROW
(a slight performance improvement).
7. Removing the code in sp_instr_set::exec_core() that sets
a variable to NULL if the value evaluation failed.
sp_rcontext::set_variable() now makes sure to reset
the variable properly by effectively calling sp_eval_expr(),
which calls virtual Field::sp_prepare_and_store_item().
Removing the similar code from sp_instr_set_row_field::exec_core()
and sp_instr_set_row_field_by_name::exec_core().
Removing the method sp_rcontext::set_variable_row_field_to_null(),
as it's not used any more.
8. Removing the call for sp_prepare_func_item() from
sp_rcontext::set_variable_row_field(), as it was duplicate:
it was done inside sp_eval_expr(). Now it's done inside
virtual Field::sp_prepare_and_store_item().
9. Moving the code from sp_instr_set_row_field_by_name::exec_core()
into sp_rcontext::set_variable_row_field_by_name(), for symmetry
with other sp_instr_set*::exec_core()/sp_rcontext::set_variable*() pairs.
Now sp_instr_set_row_field_by_name::exec_core() calls
sp_rcontext::set_variable_row_field_by_name().
10. Misc:
- Adding a helper private method sp_rcontext::virtual_tmp_table_for_row(),
reusing it in a new sp_rcontext methods.
- Removing Item_field_row::get_row_field(), as it's not used any more.
- Removing the "Item *result_item" from sp_eval_expr(),
as it's not needed any more.
This commit implements aggregate stored functions. The basic idea behind
the feature is:
* Implement a special instruction FETCH GROUP NEXT ROW that will pause
the execution of the stored function. When the instruction is reached,
execution of the initial query resumes "as if" the function returned.
This gives the server the opportunity to advance to the next row in the
result set.
* Stored aggregates behave like regular aggregate functions. The
implementation of thus resides in the class Item_sum_sp. Because it is
an aggregate function, for each new row in the group, the
Item_sum_sp::add() method will be called. This is when execution resumes
and the function does another iteration to "add" one extra element to
the final result.
* When the end of group is reached, val_xxx() method will be called for
the item. This case is handled by another execute step for the stored
function, only with a special flag to force a call to the return
handler. See Item_sum_sp::execute() for details.
To allow this pause and resume semantic, we must preserve the function
context across executions. This is stored in Item_sp::sp_query_arena only for
aggregate stored functions, but has no impact for regular functions.
We also enforce aggregate functions to include the "FETCH GROUP NEXT ROW"
instruction.
Signed-off-by: Vicențiu Ciorbaru <vicentiu@mariadb.org>
In preparation for implementing custom aggregate functions, refactor
the common code between regular stored functions and aggregate stored
functions. This includes:
* initialising SP result field
* executing a SP
* access checks
In addition, refactor sp_head::execute_function to take two extra
parameters, a function rcontext and a Query_arena. These two paremeters
were initially initialised and destroyed within
sp_head::execute_function, but for aggregate stored functions we will
require control over their lifetime. The owner of these objects now
becomes Item_sp.
Signed-off-by: Vicențiu Ciorbaru <vicentiu@mariadb.org>
