Commit graph

2415 commits

Author SHA1 Message Date
Marko Mäkelä
683fbced6b Merge 11.0 into 11.1 2024-03-28 12:15:36 +02:00
Marko Mäkelä
fec2fd6add Merge 10.11 into 11.0 2024-03-28 10:51:36 +02:00
Marko Mäkelä
788953463d Merge 10.6 into 10.11
Some fixes related to commit f838b2d799 and
Rows_log_event::do_apply_event() and Update_rows_log_event::do_exec_row()
for system-versioned tables were provided by Nikita Malyavin.
This was required by test versioning.rpl,trx_id,row.
2024-03-28 09:16:57 +02:00
Monty
f838b2d799 MDEV-33623 Partitioning is broken on big endian architectures
MDEV-33502 Slowdown when running nested statement with many partitions
caused this error as I failed to take into account bigendian architectures.

This patch also introduces bitmap_import() and bitmap_export() to be used
when one wants to store bitmaps in files/logs in a portable way.

Reviewed-by: Kristian Nielsen <knielsen@knielsen-hq.org>
2024-03-08 15:25:12 +02:00
Monty
b5d65fc105 Optimize performance of my_bitmap
MDEV-33502 Slowdown when running nested statement with many partitions

This change was triggered to help some MariaDB users with close to
10000 bits in their bitmaps.

- Change underlaying storage to be 64 bit instead of 32bit.
  - This reduses number of loops to scan bitmaps.
  - This can cause some bitmaps to be 4 byte large.
- Ensure that all not used top-bits are always 0 (simplifes code as
  the last 64 bit storage is not a special case anymore).
- Use my_find_first_bit() to find the first set bit which is much faster
  than scanning trough things byte by byte and then bit by bit.

Other things:
- Added a bool to remember if my_bitmap_init() did allocate the bitmap
  array. my_bitmap_free() will only free arrays it did allocate.
  This allowed me to remove setting 'bitmap=0' before calling
  my_bitmap_free() for cases where the bitmap's where allocated externally.
- my_bitmap_init() sets bitmap to 0 in case of failure.
- Added 'universal' asserts to most bitmap functions.
- Change all remaining calls to bitmap_init() to my_bitmap_init().
  - To finish the change from 2014.
- Changed all usage of uint32 in my_bitmap.h to my_bitmap_map.
- Updated bitmap_copy() to handle bitmaps of different size.
- Removed const from bitmap_exists_intersection() as this caused casts
  on all usage.
- Removed not used function bitmap_set_above().
- Renamed create_last_word_mask() to create_last_bit_mask() (to match
  name changes in my_bitmap.cc)
- Extended bitmap-t with test for more bitmap functions.
2024-02-27 14:51:33 +02:00
Marko Mäkelä
d73baa402a Merge 10.11 into 11.0 2024-02-20 12:02:01 +02:00
Marko Mäkelä
86c2c89743 Merge 10.6 into 10.11 2024-02-08 15:04:46 +02:00
Marko Mäkelä
91a2192bf2 Merge 10.5 into 10.6 2024-02-07 13:51:03 +02:00
Sergei Petrunia
5972f5c23b MDEV-33314: Crash in calculate_cond_selectivity_for_table() with many columns
Variant#3: moved the logic out of create_key_parts_for_pseudo_indexes

Range Analyzer (get_mm_tree functions) can only process up to MAX_KEY=64
indexes. The problem was that calculate_cond_selectivity_for_table used
it to estimate selectivities for columns, and since a table can
have > MAX_KEY columns, would invoke Range Analyzer with more than MAX_KEY
"pseudo-indexes".

Fixed by making calculate_cond_selectivity_for_table() to run Range
Analyzer with at most MAX_KEY pseudo-indexes. If there are more
columns to process, Range Analyzer will be invoked multiple times.

Also made this change:
-    param.real_keynr[0]= 0;
+    MEM_UNDEFINED(&param.real_keynr, sizeof(param.real_keynr));

Range Analyzer should have no use on real_keynr when it is run with
pseudo-indexes.
2024-02-02 22:20:10 +03:00
Sergei Golubchik
7a5448f8da Merge branch '11.0' into 11.1 2023-12-19 20:11:54 +01:00
Sergei Golubchik
8c8bce05d2 Merge branch '10.11' into 11.0 2023-12-19 15:53:18 +01:00
Sergei Golubchik
fd0b47f9d6 Merge branch '10.6' into 10.11 2023-12-18 11:19:04 +01:00
Sergei Golubchik
e95bba9c58 Merge branch '10.5' into 10.6 2023-12-17 11:20:43 +01:00
Alexander Barkov
4ced4898fd MDEV-32958 Unusable key notes do not get reported for some operations
Enable unusable key notes for non-equality predicates:
   <, <=, =>, >, BETWEEN, IN, LIKE

Note, in some scenarios it displays duplicate notes, e.g.
for queries with ORDER BY:

  SELECT * FROM t1
  WHERE    indexed_string_column >= 10
  ORDER BY indexed_string_column
  LIMIT 5;

This should be tolarable. Getting rid of the diplicate note
completely would need a much more complex patch, which is
not desiable in 10.6.

Details:

- Changing RANGE_OPT_PARAM::note_unusable_keys from bool
  to a new data type Item_func::Bitmap, so the caller can
  choose with a better granuality which predicates
  should raise unusable key notes inside the range optimizer:
    a. all predicates (=, <=>, <, <=, =>, >, BETWEEN, IN, LIKE)
    b. all predicates except equality (=, <=>)
    c. none of the predicates

  "b." is needed because in some scenarios equality predicates (=, <=>)
  send unusable key notes at an earlier stage, before the range optimizer,
  during update_ref_and_keys(). Calling the range optimizer with
  "all predicates" would produce duplicate notes for = and <=> in such cases.

- Fixing get_quick_record_count() to call the range optimizer
  with "all predicates except equality" instead of "none of the predicates".
  Before this change the range optimizer suppressed all notes for
  non-equality predicates: <, <=, =>, >, BETWEEN, IN, LIKE.
  This actually fixes the reported problem.

- Fixing JOIN::make_range_rowid_filters() to call the range optimizer
  with "all predicates except equality" instead of "all predicates".
  Before this change the range optimizer produced duplicate notes
  for = and <=> during a rowid_filter optimization.

- Cleanup:
  Adding the op_collation argument to Field::raise_note_cannot_use_key_part()
  and displaying the operation collation rather than the argument collation
  in the unusable key note. This is important for operations with more than
  two arguments: BETWEEN and IN, e.g.:

    SELECT * FROM t1
    WHERE column_utf8mb3_general_ci
          BETWEEN 'a' AND 'b' COLLATE utf8mb3_unicode_ci;

    SELECT * FROM t1
    WHERE column_utf8mb3_general_ci
          IN ('a', 'b' COLLATE utf8mb3_unicode_ci);

    The note for 'a' now prints utf8mb3_unicode_ci as the collation.
    which is the collation of the entire operation:

      Cannot use key key1 part[0] for lookup:
      "`column_utf8mb3_general_ci`" of collation `utf8mb3_general_ci` >=
      "'a'" of collation `utf8mb3_unicode_ci`

    Before this change it printed the collation of 'a',
    so the note was confusing:

      Cannot use key key1 part[0] for lookup:
      "`column_utf8mb3_general_ci`" of collation `utf8mb3_general_ci` >=
      "'a'" of collation `utf8mb3_general_ci`"
2023-12-11 08:55:27 +04:00
Yuchen Pei
13dd787530
Merge branch '10.4' into 10.5 2023-12-07 16:38:00 +11:00
Oleg Smirnov
a8bd6a9813 MDEV-15656 Assertion `is_last_prefix <= 0' failed in QUICK_GROUP_MIN_MAX_SELECT::get_next
When QUICK_GROUP_MIN_MAX_SELECT is initialized or being reset
it stores the prefix of the last group of the index chosen for
retrieving data (last_value). Later, when looping through records
at get_next() method, the server checks whether the retrieved
group is the last, and if so, it finishes processing.

At the same time, it looks like there is no need for that additional
check since method next_prefix() returns HA_ERR_KEY_NOT_FOUND
or HA_ERR_END_OF_FILE when there are no more satisfying records.
If we do not perform the check, we do not need to retrieve and
store last_value either.

This commit removes using of last_value from QUICK_GROUP_MIN_MAX_SELECT.

Reviewer: Sergei Petrunia <sergey@mariadb.com>
2023-12-06 15:19:49 +03:00
Oleksandr Byelkin
0f5613a25f Merge branch '11.0' into 11.1 2023-11-08 18:03:08 +01:00
Oleksandr Byelkin
48af85db21 Merge branch '10.11' into 11.0 2023-11-08 17:09:44 +01:00
Oleksandr Byelkin
04d9a46c41 Merge branch '10.6' into 10.10 2023-11-08 16:23:30 +01:00
Monty
6f091434f3 MDEV-32531 MSAN / Valgrind errors in Item_func_like::get_mm_leaf with temporal field
Added missing initializer
2023-10-30 14:44:26 +02:00
Rex
ec2574fd8f MDEV-31983 jointable materialization subquery optimization ignoring
...errors, then failing ASSERT.

UPDATE queries treat warnings as errors. In this case, an invalid
condition "datetime_key_col >= '2012-01'" caused warning-as-error inside
SQL_SELECT::test_quick_select().

The code that called test_quick_select() ignored this error and continued
join optimization. Then it eventually reached a thd->is_error() check
and failed to setup SJ-Materialization which failed an assert.

Fixed this by making SQL_SELECT::test_quick_select() return error in
its return value, and making any code that calls it to check for error
condition and abort the query if the error is returned.

Places in the code that didn't check for errors from
SQL_SELECT::test_quick_select but now do:
- get_quick_record_count() call in make_join_statistics(),
- test_if_skip_sort_order(),
- "Range checked for each record" code.

Extra error handling fixes and commit text wording by Sergei Petrunia,

Reviewed-by: Sergei Petrunia, Oleg Smirnov
2023-10-25 17:21:37 +03:00
Marko Mäkelä
5a8fca5a4f Merge 10.6 into 10.10 2023-10-23 18:43:36 +03:00
Sergei Petrunia
4941ac9192 MDEV-32113: utf8mb3_key_col=utf8mb4_value cannot be used for ref
(Variant#3: Allow cross-charset comparisons, use a special
CHARSET_INFO to create lookup keys. Review input addressed.)

Equalities that compare utf8mb{3,4}_general_ci strings, like:

  WHERE ... utf8mb3_key_col=utf8mb4_value    (MB3-4-CMP)

can now be used to construct ref[const] access and also participate
in multiple-equalities.
This means that utf8mb3_key_col can be used for key-lookups when
compared with an utf8mb4 constant, field or expression using '=' or
'<=>' comparison operators.

This is controlled by optimizer_switch='cset_narrowing=on', which is
OFF by default.

IMPLEMENTATION
Item value comparison in (MB3-4-CMP) is done using utf8mb4_general_ci.
This is valid as any utf8mb3 value is also an utf8mb4 value.

When making index lookup value for utf8mb3_key_col, we do "Charset
Narrowing": characters that are in the Basic Multilingual Plane (=BMP) are
copied as-is, as they can be represented in utf8mb3. Characters that are
outside the BMP cannot be represented in utf8mb3 and are replaced
with U+FFFD, the "Replacement Character".

In utf8mb4_general_ci, the Replacement Character compares as equal to any
character that's not in BMP. Because of this, the constructed lookup value
will find all index records that would be considered equal by the original
condition (MB3-4-CMP).

Approved-by: Monty <monty@mariadb.org>
2023-10-19 17:24:30 +03:00
Marko Mäkelä
9b2a65e41a Merge 11.0 into 11.1 2023-10-19 08:26:16 +03:00
Marko Mäkelä
be24e75229 Merge 10.11 into 11.0 2023-10-19 08:12:16 +03:00
Marko Mäkelä
d5e15424d8 Merge 10.6 into 10.10
The MDEV-29693 conflict resolution is from Monty, as well as is
a bug fix where ANALYZE TABLE wrongly built histograms for
single-column PRIMARY KEY.
Also includes a fix for safe_malloc error reporting.

Other things:
- Copied main.log_slow from 10.4 to avoid mtr issue

Disabled test:
- spider/bugfix.mdev_27239 because we started to get
  +Error	1429 Unable to connect to foreign data source: localhost
  -Error	1158 Got an error reading communication packets
- main.delayed
  - Bug#54332 Deadlock with two connections doing LOCK TABLE+INSERT DELAYED
    This part is disabled for now as it fails randomly with different
    warnings/errors (no corruption).
2023-10-14 13:36:11 +03:00
Monty
8bf17c579b MDEV-32388 MSAN / Valgrind errors in Item_func_like::get_mm_leaf upon query from partitioned table
The problem was that RANGE_OPT_PARAM was not completely initialized in
some cases.
Added bzero() to ensure that all elements are always initialized.
2023-10-10 14:05:02 +03:00
Monty
d4347177c7 Change SEL_ARG::MAX_SEL_ARGS to a user defined variable optimizer_max_sel_args
This allows a user to to change the default value of MAX_SEL_ARGS (16000)
in the rare case where they neeed more generated SEL_ARGS (as part of
the range optimizer)
2023-10-03 08:25:31 +03:00
Monty
4e9322e2ff MDEV-32203 Raise notes when an index cannot be used on data type mismatch
Raise notes if indexes cannot be used:
- in case of data type or collation mismatch (diferent error messages).
- in case if a table field was replaced to something else
  (e.g. Item_func_conv_charset) during a condition rewrite.

Added option to write warnings and notes to the slow query log for
slow queries.

New variables added/changed:

- note_verbosity, with is a set of the following options:
  basic            - All old notes
  unusable_keys    - Print warnings about keys that cannot be used
                     for select, delete or update.
  explain          - Print unusable_keys warnings for EXPLAIN querys.

The default is 'basic,explain'. This means that for old installations
the only notable new behavior is that one will get notes about
unusable keys when one does an EXPLAIN for a query. One can turn all
of all notes by either setting note_verbosity to "" or setting sql_notes=0.

- log_slow_verbosity has a new option 'warnings'. If this is set
  then warnings and notes generated are printed in the slow query log
  (up to log_slow_max_warnings times per statement).

- log_slow_max_warnings   - Max number of warnings written to
                            slow query log.

Other things:
- One can now use =ALL for any 'set' variable to set all options at once.
  For example using "note_verbosity=ALL" in a config file or
  "SET @@note_verbosity=ALL' in SQL.
- mysqldump will in the future use @@note_verbosity=""' instead of
  @sql_notes=0 to disable notes.
- Added "enum class Data_type_compatibility" and changing the return type
  of all Field::can_optimize*() methods from "bool" to this new data type.

Reviewer & Co-author: Alexander Barkov <bar@mariadb.com>
- The code that prints out the notes comes mainly from Alexander
2023-10-03 08:25:31 +03:00
Oleksandr Byelkin
f5fae75652 Merge branch '11.0' into 11.1 2023-08-09 08:25:14 +02:00
Oleksandr Byelkin
51f9d62005 Merge branch '10.11' into 11.0 2023-08-09 07:53:48 +02:00
Oleksandr Byelkin
34a8e78581 Merge branch '10.6' into 10.9 2023-08-04 08:01:06 +02:00
Oleksandr Byelkin
6bf8483cac Merge branch '10.5' into 10.6 2023-08-01 15:08:52 +02:00
Oleksandr Byelkin
65405308a1 Merge branch '10.4' into 10.5 2023-08-01 11:52:13 +02:00
Sergei Golubchik
4dd38f9f39 MDEV-31800 Problem with open ranges on prefix blobs keys
don't construct open ranges from prefix blob keys for < (less than)
just as it's already done for > (greater than)

because prefix KEY_PART doesn't create prefix Field for blobs
(see open_table_from_share() near "Create a new field for the key part"),
so stored_field_cmp_to_item() will compare the original field to the
value not taking the prefix length into account.
2023-07-31 22:46:47 +02:00
Oleksandr Byelkin
f52954ef42 Merge commit '10.4' into 10.5 2023-07-20 11:54:52 +02:00
Rex
f17a865c39 MDEV-30710 Incorrect operator when comparing large unsigned integers.
When constructing a SEL_TREE, an unsigned integer greater than
its signed equivalent caused an incorrect comparison operator to
be chosen.
2023-07-19 15:59:33 +11:00
Marko Mäkelä
8290a46d50 Merge 11.0 into 11.1 2023-06-28 09:38:59 +03:00
Monty
f25a74c0b0 Fixed typo on opt_range.cc: SEL_ARG::number_of_eq_groups()
It could cause wrong range estimation for GROUP BY queries that are
using 'WHERE index_part >= constant'.
(The function was trying to check for 'index_part = constant')

Reporter: Yuty Chaikou
2023-06-18 12:11:18 +03:00
Marko Mäkelä
3883eb63dc Merge 11.0 into 11.1 2023-06-08 14:09:21 +03:00
Sergei Petrunia
f5dceafd0b MDEV-30964: MAX_SEL_ARG memory exhaustion is not visible in the optimizer trace
Add printing
2023-06-08 14:02:34 +03:00
Marko Mäkelä
223c2c5b9d Merge 10.6 into 10.9 2023-06-08 10:46:19 +03:00
Marko Mäkelä
80585c9d6f Merge 10.5 into 10.6 2023-06-08 10:42:56 +03:00
Monty
ded4ed3220 MDEV-30944 Range_rowid_filter::fill() leaves file->keyread at MAX_KEY
This test case exposed 2 different bugs:
- When replacing a range with an index scan on a covering key
  in test_if_skip_sort_order() we didn't disable filtering.
  Filtering does not make much sense in this case.
  - Fixed by disabling filtering in this case.
- Range_rowid_filter::fill() did not take into account that keyread
  could already active, which caused an assert when it tried to
  activate another keyread.
  - Fixed by remembering old keyread state at start and restoring it
    at end.

Other things:
- ha_start_keyread() allowed multiple calls. This is wrong, especially
  as we do no check if the index changed!
  I added an assert() to ensure that we don't call it there is already
  an active keyread.
- ha_end_keyread() always called ha_extra(), even if keyread was not
  active.  Added a check to avoid the extra call.
2023-06-07 18:44:12 +03:00
Monty
07b02ab40e MDEV-31356: Range cost calculations does not take into account join_buffer
This patch also fixes
MDEV-31391 Assertion `((best.records_out) == 0.0 ... failed

Cost changes caused by this change:
- range queries with join buffer now have a notable smaller cost.
- range ranges are bit more expensive as the MULTI_RANGE_COST is now
  properly applied to it in all cases (this extra cost is equal to a
  key lookup).
- table scan cost is slight smaller as we now assume data is cached in
  the engine after the first scan pass. (We did this before for range
  scans and other access methods).
- partition tables had wrong values for max_row_blocks and
  max_index_blocks.  Correcting this, causes range access on
  partitioned tables to have slightly higher cost because of the
  increased estimated IO.
- Using first match + join buffer caused 'filtered' to be calcualted
  wrong.  (Only affected EXPLAIN, not query costs).
- Added cost_without_join_buffer to optimizer_trace.
- check_quick_select() adjusted the number of rows according to persistent
  statistics, but did not adjust cost. Now fixed.

The big change in the patch are:

- In best_access_path(), where we now are using storing the cost in
  'ALL_READ_COST cost' and only converting it to a double at the end.
   This allows us to more exactly calculate the effect of the join_cache.
- In JOIN_TAB::estimate_scan_time(), store the cost also in a
  ALL_READ_COST object.

One of effect if this change is that when joining very small tables:

t1    some_access_method
t2    range
t3    ALL         Use join buffer

This is swiched to

t1      some_access_method
t3      ALL
t2      range      use join buffer

Both plans has the same cost, but as table scan in this case has less
cost than rang, the table scan will be considered first and thus have
precidence.

Test case changes:
- optimizer_trace          - Addition of cost_without_join_buffer
- subselect_mat_cost_bugs  - Small tables and scan versus range
- range & range_mrr_icp    - Range + join_cache is faster than ref
- optimizer_trace          - cost_without_join_buffer, smaller scan cost,
                             range setup cost.
- mrr                      - range+join_buffer used as smaller cost
2023-06-07 18:42:58 +03:00
Sergei Petrunia
a0e7bd735b MDEV-31380: Assertion `s->table->opt_range_condition_rows <= s->found_records' failed
LooseScan code set opt_range_condition_rows to be the

  MIN(loose_scan_plan->records, table->records)

totally ignoring possible quick range selects.  If there was a quick
select $QUICK on another index with

  $QUICK->records < loose_scan_plan->records

this would create a situation where

   opt_range_condition_rows > $QUICK->records

which causes an assert in 10.6+ and potentially wrong query plan
choice in 10.5.

Fixed by making opt_range_condition_rows to be the minimum #rows
of any quick select.

Approved-by: Monty <monty@mariadb.org>
2023-06-07 13:54:34 +03:00
Sergei Golubchik
cbabb95915 Merge branch '11.0' into 11.1 2023-06-05 20:15:15 +02:00
Marko Mäkelä
54819192fe Merge 10.11 into 11.0 2023-04-26 18:50:15 +03:00
Marko Mäkelä
c15c8ef3e3 Merge 10.6 into 10.8 2023-04-26 13:58:40 +03:00
Marko Mäkelä
818d5e4814 Merge 10.5 into 10.6 2023-04-25 13:10:33 +03:00