This is MDEV-7601, including it's sub tasks MDEV-7594, MDEV-7555, MDEV-7590, MDEV-7581, MDEV-7589
The problem was that select_lex->non_agg_fields was not properly reset for re-execution and this caused an overwrite of a random memory position.
The fix was move non_agg_fields from select_lext to JOIN, which is properly reset.
~40% bugfixed(*) applied
~40$ bugfixed reverted (incorrect or we're not buggy)
~20% bugfixed applied, despite us being not buggy
(*) only changes in the server code, e.g. not cmakefiles
ISSUE:
------
For UNION of selects, rows examined by the query will be sum
of rows examined by individual select operations and rows
examined for union operation. The value of session level
global counter that is used to count the rows examined by a
select statement should be accumulated and reset before it
is used for next select statement. But we have missed to
reset the same. Because of this examined row count of a
select query is accounted more than once.
SOLUTION:
---------
In union reset the session level global counter used to
accumulate count of examined rows after its value is saved.
mysql-test/r/union.result:
Expected output of testcase added.
mysql-test/t/union.test:
Test to verify examined row count of Union operations.
sql/sql_union.cc:
Reset the value of thd->examined_row_count after
accumulating the value.
WHERE ONE OF SELECT* IS DISTINCT FAILS.
ISSUE:
------
There are 2 issues related to explain union.
1. If we have subquery with union of selects. And, one of
the select need temp table to materialize its results
then it will replace its query structure with a simple
select from temporary table. Trying to display new
internal temporary table scan resulted in crash. But to
display the query plan, we should save the original
query structure.
2. Multiple execution of prepared explain statement which
have union of subqueries resulted in crash. If we have
constant subqueries, fake select used in union operation
will be evaluated once before using it for explain.
During first execution we have set fake select options to
SELECT_DESCRIBE, but did not reset after the explain.
Hence during next execution of prepared statement during
first time evaluation of fake select we had our select
options as SELECT_DESCRIBE this resulted in improperly
initialized data structures and crash.
SOLUTION:
---------
1. If called by explain now we save the original query
structure. And this will be used for displaying.
2. Reset the fake select options after it is called for
explain of union.
sql/sql_select.cc:
Reset the fake select options after it is called for explain of union
sql/sql_union.cc:
If called by explain but not from select_describe and we
need a temp table, then we create a temp join to preserve
original query structure.
Analysis:
st_select_lex_unit::prepare() computes can_skip_order_by as TRUE.
As a result join->prepare() gets called with order == NULL, and
doesn't do name resolution for the inner ORDER clause. Due to this
the prepare phase doesn't detect that the query references non-exiting
function and field.
Later join->optimize() calls update_used_tables() for a non-resolved
Item_field, which understandably has no Field object. This call results
in a crash.
Solution:
Resolve unnecessary ORDER BY clauses to detect if they reference non-exising
objects. Then remove such clauses from the JOIN object.
ORDER BY does not work
Use "dynamic" row format (instead of "block") for MARIA internal
temporary tables created for cursors.
With "block" row format MARIA may shuffle rows, with "dynamic" row
format records are inserted sequentially (there are no gaps in data
file while we fill temporary tables).
This is needed to preserve row order when scanning materialized cursors.
Apply the patch from Patryk Pomykalski:
- create_internal_tmp_table_from_heap() will now return information whether
the last row that we tried to write was a duplicate row.
(mysql-5.6 also has this change)
KILL now breaks locks inside InnoDB
Fixed possible deadlock when running INNODB STATUS
Added ha_kill_query() and kill_query() to send kill signal to all storage engines
Added reset_killed() to ensure we don't reset killed state while awake() is getting called
include/mysql/plugin.h:
Added thd_mark_as_hard_kill()
include/mysql/plugin_audit.h.pp:
Added thd_mark_as_hard_kill()
include/mysql/plugin_auth.h.pp:
Added thd_mark_as_hard_kill()
include/mysql/plugin_ftparser.h.pp:
Added thd_mark_as_hard_kill()
sql/handler.cc:
Added ha_kill_query() to send kill signal to all storage engines
sql/handler.h:
Added ha_kill_query() and kill_query() to send kill signal to all storage engines
sql/log_event.cc:
Use reset_killed()
sql/mdl.cc:
use thd->killed instead of thd_killed() to abort on soft kill
sql/sp_rcontext.cc:
Use reset_killed()
sql/sql_class.cc:
Fixed possible deadlock in INNODB STATUS by not getting thd->LOCK_thd_data if it's locked.
Use reset_killed()
Tell storge engines that KILL has been sent
sql/sql_class.h:
Added reset_killed() to ensure we don't reset killed state while awake() is getting called.
Added mark_as_hard_kill()
sql/sql_insert.cc:
Use reset_killed()
sql/sql_parse.cc:
Simplify detection of killed queries.
Use reset_killed()
sql/sql_select.cc:
Use reset_killed()
sql/sql_union.cc:
Use reset_killed()
storage/innobase/handler/ha_innodb.cc:
Added innobase_kill_query()
Fixed error reporting for interrupted queries.
storage/xtradb/handler/ha_innodb.cc:
Added innobase_kill_query()
Fixed error reporting for interrupted queries.
The problem was that was_null and null_value variables was reset in each reexecution of IN subquery, but engine rerun only for non-constant subqueries.
Fixed checking constant in Item_equal sort.
Fix constant reporting in Item_subselect.
https://mariadb.atlassian.net/browse/MDEV-28
This task implements a new clause LIMIT ROWS EXAMINED <num>
as an extention to the ANSI LIMIT clause. This extension
allows to limit the number of rows and/or keys a query
would access (read and/or write) during query execution.
The result of materialization of the right part of an IN subquery predicate
is placed into a temporary table. Each row of the materialized table is
distinct. A unique key over all fields of the temporary table is defined and
created. It allows to perform key look-ups into the table.
The table created for a materialized subquery can be accessed by key as
any other table. The function best_access-path search for the best access
to join a table to a given partial join. With some where conditions this
function considers a possibility of a ref_or_null access. If such access
employs the unique key on the temporary table then when estimating
the cost this access the function tries to use the array rec_per_key. Yet,
such array is not built for this unique key. This causes a crash of the server.
Rows returned by the subquery that contain nulls don't have to be placed
into temporary table, as they cannot be match any row produced by the
left part of the subquery predicate. So all fields of the temporary table
can be defined as non-nullable. In this case any ref_or_null access
to the temporary table does not make any sense and it does not make sense
to estimate such an access.
The fix makes sure that the temporary table for a materialized IN subquery
is defined with columns that are all non-nullable. The also ensures that
any row with nulls returned by the subquery is not placed into the
temporary table.
Problem was that now we can merge derived table (subquery in the FROM clause).
Fix: in case of detected conflict and presence of derived table "over" the table which cased the conflict - try materialization strategy.
sql/sql_insert.cc:
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
******
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
sql/sql_table.cc:
small cleanup
******
small cleanup
Reset 'examined_rows_count' in union to not count same rows twice
mysql-test/r/subselect_mat_cost.result:
Test also slow query logging
mysql-test/t/subselect_mat_cost.test:
Test also slow query logging
sql/sql_union.cc:
Reset 'examined_rows_count' in union to not count same rows twice
We must allocate a larger ref_pointer_array. We failed to account for extra
items allocated here:
#0 find_order_in_list
uint el= all_fields.elements;
all_fields.push_front(order_item); /* Add new field to field list. */
ref_pointer_array[el]= order_item;
order->item= ref_pointer_array + el;
#1 setup_order
#2 setup_without_group
#3 JOIN::prepare
mysql-test/r/order_by.result:
New test case.
mysql-test/r/union.result:
New test case.
mysql-test/t/order_by.test:
New test case.
mysql-test/t/union.test:
New test case.
sql/sql_lex.cc:
find_order_in_list() may need some extra space, so multiply og_num by two.
sql/sql_union.cc:
For UNION, the 'n_sum_items' are accumulated in the "global_parameters" select_lex.
This number must be propagated to setup_ref_array()
When preparing a 'fake_select_lex' we need to use global_parameters->order_list
rather than fake_select_lex->order_list (see comments inside st_select_lex_unit::cleanup)
Split status variable Rows_read to Rows_read and Rows_tmp_read so that one can see how much real data is read.
Same was done with with Handler_update and Handler_write.
Fixed bug in MEMORY tables where some variables was counted twice.
Added new internal handler call 'ha_close()' to have one place to gather statistics.
Fixed bug where thd->open_options was set to wrong value when doing admin_recreate_table()
mysql-test/r/status.result:
Updated test results and added new tests
mysql-test/r/status_user.result:
Udated test results
mysql-test/t/status.test:
Added new test for temporary table status variables
sql/ha_partition.cc:
Changed to call ha_close() instead of close()
sql/handler.cc:
Added internal_tmp_table variable for easy checking of temporary tables.
Added new internal handler call 'ha_close()' to have one place to gather statistics.
Gather statistics for internal temporary tables.
sql/handler.h:
Added handler variables internal_tmp_table, rows_tmp_read.
Split function update_index_statistics() to two.
Added ha_update_tmp_row() for faster tmp table handling with more statistics.
sql/item_sum.cc:
ha_write_row() -> ha_write_tmp_row()
sql/multi_range_read.cc:
close() -> ha_close()
sql/mysqld.cc:
New status variables: Rows_tmp_read, Handler_tmp_update and Handler_tmp_write
sql/opt_range.cc:
close() -> ha_close()
sql/sql_base.cc:
close() -> ha_close()
sql/sql_class.cc:
Added handling of rows_tmp_read
sql/sql_class.h:
Added new satistics variables.
rows_read++ -> update_rows_read() to be able to correctly count reads to internal temp tables.
Added handler::ha_update_tmp_row()
sql/sql_connect.cc:
Added comment
sql/sql_expression_cache.cc:
ha_write_row() -> ha_write_tmp_row()
sql/sql_select.cc:
close() -> ha_close()
ha_update_row() -> ha_update_tmp_row()
sql/sql_show.cc:
ha_write_row() -> ha_write_tmp_row()
sql/sql_table.cc:
Fixed bug where thd->open_options was set to wrong value when doing admin_recreate_table()
sql/sql_union.cc:
ha_write_row() -> ha_write_tmp_row()
sql/sql_update.cc:
ha_write_row() -> ha_write_tmp_row()
sql/table.cc:
close() -> ha_close()
storage/heap/ha_heap.cc:
Removed double counting of statistic variables.
close -> ha_close() to get tmp table statistics.
storage/maria/ha_maria.cc:
close -> ha_close() to get tmp table statistics.
The patch imposes unconditional materialization for derived tables
used in update and multi-update statements.
Fixed a bug with a wrong order of processing derived tables/views
at the prepare stage that caused a crash for the variant of the
query from test case for bug 52157.
Resolved all conflicts, bad merges and fixed a few minor bugs in the code.
Commented out the queries from multi_update, view, subselect_sj, func_str,
derived_view, view_grant that failed either with crashes in ps-protocol or
with wrong results.
The failures are clear indications of some bugs in the code and these bugs
are to be fixed.
