The patch actually fixes the old defect of the optimizer that
could not extract keys for range access from IN predicates
with row arguments.
This problem was resolved in the mysql-5.7 code. The patch
supersedes what was done there:
- it can build range access when not all components of
the first row argument are refer to the columns of the table
for which the range access is constructed.
- it can use equality predicates to build range access
to the table that is not referred to in this argument.
In get_mm_tree we have to change Field_geom::geom_type to
GEOMETRY as we have to let storing all types of the spatial features
in the field. So now we restore the original geom_type as it's
done.
Also, implement MDEV-11027 a little differently from 5.5 and 10.0:
recv_apply_hashed_log_recs(): Change the return type back to void
(DB_SUCCESS was always returned).
Report progress also via systemd using sd_notifyf().
* rename to "keyread" (to avoid conflicts with tokudb),
* change from bool to uint and store the keyread index number there
* provide a bool accessor to check if keyread is enabled
mark_columns_used_by_index used to do
reset + mark_columns_used_by_index_no_reset + start keyread + set bitmaps
Now prepare_for_keyread does that, while mark_columns_used_by_index
does only reset + mark_columns_used_by_index_no_reset,
just as its name suggests.
move TABLE::key_read into handler. Because in index merge and DS-MRR
there can be many handlers per table, and some of them use
key read while others don't. "keyread" is really per handler,
not per TABLE property.
When building different range and index-merge trees the range optimizer
could build an index-merge tree with an index scan containing less ranges
then needed. This index-merge could be chosen as the best. Following this
index-merge the executioner missed some rows in the result set.
The invalid index scan was built due to an inconsistency in the code
back-ported from mysql into 5.3 that fixed mysql bug #11765831:
the code added to key_or() could change shared keys of the second
ored tree. Partially the problem was fixed in the patch for mariadb
bug #823301, but it turned out that only partially.
Found and fixed 2 problems:
- Filesort addon fields didn't mark virtual columns properly
- multi-range-read calculated vcol bitmap but was not using it.
This caused wrong vcol field to be calculated on read, which caused the assert.
In file sql/opt_range.cc,when calculate_cond_selectivity_for_table() is called with optimizer_use_condition_selectivity=4 then
- thd->no_errors is set to 1
- the original value of thd->no_error is not restored to its original value
- this is causing the assertion to fail in the subsequent queries
Fixed by restoring the original value of thd->no_errors
In the function create_key_parts_for_pseudo_indexes()
the key part structures of pseudo-indexes created for
BLOB fields were set incorrectly.
Also the key parts for long fields must be 'truncated'
up to the maximum length acceptable for key parts.
Implement a technique mentioned in the MDEV. Under certain conditions,
cond(inner_table.col) can be substituted for cond(outer_table.col) for
the purpose of range analysis.
Fix get_quick_keys(): When building range tree from a condition
in form
keypart1=const AND (keypart2 < 0 OR keypart2>=0)
the SEL_ARG for keypart2 represents an interval (-inf, +inf).
However, the logic that sets UNIQUE_RANGE flag fails to recognize
this, and sets UNIQUE_RANGE flag if (keypart1, keypart2) covered
a unique key.
As a result, range access executor assumes the interval can have
at most one row and only reads the first row from it.
The crash was caused by this problem:
get_best_group_min_max() tries to construct query plans for keys that
are not processed by the range optimizer. This wasn't a problem as long
as SEL_TREE::keys was an array of MAX_KEY elements.
However, now it is a Mem_root_array and only has elements for the used
keys, and get_best_group_min_max attempts to address beyond the end of
the array.
The obvious way to fix the crash was to port (and improve) a part of
96fcfcbd7b5120e8f64fd45985001eca8d36fbfb from mysql-5.7. This makes
get_best_group_min_max not to consider indexes that Mem_root_arrays
have no element for.
After that, I got non-sensical query plans (see MDEV-10325 for details).
Fixed that by making get_best_group_min_max to check if the index is in
table->keys_in_use_for_group_by bitmap.
The problem was that the loop in get_func_mm_tree()
accessed improperly initialized instances of String,
which resided in the bzero'ed part of the in_vector::base array.
Strings in in_vector::base are originally initialized
in Item_func_in::fix_length_and_dec(),
in in_vector::in_vector() using sql_calloc,
rather than using a String constructor, so their str_charset
members are originally equal to NULL.
Strings in in_vector::base are later initialized
to good values in Item_func_in::fix_length_and_dec(),
using array->set(), in this code:
uint j=0;
for (uint i=1 ; i < arg_count ; i++)
{
array->set(j,args[i]);
if (!args[i]->null_value) // Skip NULL values
j++;
else
have_null= 1;
}
if ((array->used_count= j))
array->sort();
NULLs are not taken into account, so at the end
array->used_count can be smaller than array->count.
This patch fixes the loop in opt_range.cc, in get_func_mm_tree(),
to access only properly initialized elements in in_vector::base,
preventing access to its bzero'ed non-initialized tail.
Part #2: make tree_or(tree1, tree2) to reuse tree1 for the result object
for simple cases. These include key IN (c1, ... cN).
The reuse was happening in old MySQL versions, but we stopped doing it
in the "fair choice between range and index_merge" patch.
A partial backport of 67f21fb3a077dedfd14b9ca720e926c55e682f93,
Bug#22283790: RANGE OPTIMIZER UTILIZES TOO MUCH MEMORY WITH MANY OR CONDITIONS
The backported part changes SEL_TREE::keys from being an array of
MAX_KEY elements (64*8=512 bytes) to a Mem_root_array<SEL_ARG*> (32 bytes +
alloc'ed array of as many elements as we need).
The patch doesn't fix the "not limiting memory" part, but the memory usage
is much lower with it.
filesort and init_read_record() for the same table.
This will simplify code for WINDOW FUNCTIONS (MDEV-6115)
- Filesort_info renamed to SORT_INFO and moved to filesort.h
- filesort now returns SORT_INFO
- init_read_record() now takes a SORT_INFO parameter.
- unique declaration is moved to uniques.h
- subselect caching of buffers is now more explicit than before
- filesort_buffer is now reusable even if rec_length has changed.
- filsort_free_buffers() and free_io_cache() calls are removed
- Remove one malloc() when using get_addon_fields()
Other things:
- Added --debug-assert-on-not-freed-memory option to make it easier to
debug some not-freed-memory issues.
"Re-factor the code for post-join operations".
The patch mainly contains the code ported from mysql-5.6 and
created for two essential architectural changes:
1. WL#5558: Resolve ORDER BY execution method at the optimization stage
2. WL#6071: Inline tmp tables into the nested loops algorithm
The first task was implemented for mysql-5.6 by Ole John Aske.
It allows to make all decisions on ORDER BY operation at the optimization
stage.
The second task implemented for mysql-5.6 by Evgeny Potemkin adds JOIN_TAB
nodes for post-join operations that require temporary tables. It allows
to execute these operations within the nested loops algorithm that used to
be used before this task only for join queries. Besides these task moves
all planning on the execution of these operations from the execution phase
to the optimization phase.
Some other re-factoring changes of mysql-5.6 were pulled in, mainly because
it was easier to pull them in than roll them back. In particular all
changes concerning Ref_ptr_array were incorporated.
The port required some changes in the MariaDB code that concerned the
functionality of EXPLAIN and ANALYZE. This was done mainly by Sergey
Petrunia.
create_partition_index_description() had wrong logic to calculate
length of the key value buffer that is used by the range optimizer.
For some reason it used MAX(partitioning_columns_len,
subpartitioning_columns_len) while it should use SUM of these values.
changes in query execution plans.
Fixed by introducing table->rpl_write_set which holds which columns should
be stored in the binary log.
Other things:
- Removed some not needed references to read_set and write_set to make
code really changing read_set and write_set easier to read
(in opt_range.cc)
- Added error handling of failed unpack_current_row()
- Added missing call to mark_columns_needed_for_insert() for DELAYED INSERT
- Removed not used functions in_read_set() and in_write_set()
- In rpl_record.cc, removed not used variable error
The bitmap implementation defines two template Bitmap classes. One
optimized for 64-bit (default) wide bitmaps while the other is used for
all other widths.
In order to optimize the computations, Bitmap<64> class has defined its
own member functions for bitmap operations, the other one, however,
relies on mysys' bitmap implementation (mysys/my_bitmap.c).
Issue 1:
In case of non 64-bit Bitmap class, intersect() wrongly reset the
received bitmap while initialising a new local bitmap structure
(bitmap_init() clears the bitmap buffer) thus, the received bitmap was
getting cleared.
Fixed by initializing the local bitmap structure by using a temporary
buffer and later copying the received bitmap to the initialised bitmap
structure.
Issue 2:
The non 64-bit Bitmap class had the Iterator missing which caused
compilation failure.
Also added a cmake variable to hold the MAX_INDEXES value when supplied
from the command prompt. (eg. cmake .. -DMAX_INDEXES=128U). Checks have
been put in place to trigger build failure if MAX_INDEXES value is
greater than 128.
Test modifications:
* Introduced include/have_max_indexes_[64|128].inc to facilitate
skipping of tests for which the output differs with different
MAX_INDEXES.
* Introduced include/max_indexes.inc which would get modified by cmake
to reflect the MAX_INDEXES value used to build the server. This file
simply sets an mtr variable '$max_indexes' to show the MAX_INDEXES
value, which will then be consumed by the above introduced include file.
* Some tests (portions), dependent on MAX_INDEXES value, have been moved
to separate test files.
When we calculate max_key_len for RANGE_OPT_PARAM::min_key/max_key, take
into account that QUICK_RANGE::QUICK_RANGE for some reason assumes
that there is one more byte there:
max_key((uchar*) sql_memdup(max_key_arg,max_length_arg+1)),
- Turning get_mm_tree_for_const() from a static function into
a protected method in Item.
- Adding a new class Item_bool_func2_with_rev, for the functions and operators
that have a reverse function and can use the range optimizer for
to optimize "value OP field" as "field REV_OP value". Deriving
Item_bool_rowready_func2 and Item_funt_spatial_rel from the new class.
- Removing Item_bool_func2::have_rev_func().
