Commit graph

2227 commits

Author SHA1 Message Date
Monty
009db2288b Fixed limit optimization in range optimizer
The issue was that when limit is used,
SQL_SELECT::test_quick_select would set the cost of table scan to be
unreasonable high to force a range to be used.
The problem with this approach was that range was used even when the
cost of range, when it would only read 'limit rows' would be higher
than the cost of a table scan.

This patch fixes it by not accepting ranges when the range can never
have a lower cost than a table scan, even if every row would match the
WHERE clause.
2023-02-02 23:54:57 +03:00
Monty
b66cdbd1ea Changing all cost calculation to be given in milliseconds
This makes it easier to compare different costs and also allows
the optimizer to optimizer different storage engines more reliably.

- Added tests/check_costs.pl, a tool to verify optimizer cost calculations.
  - Most engine costs has been found with this program. All steps to
    calculate the new costs are documented in Docs/optimizer_costs.txt

- User optimizer_cost variables are given in microseconds (as individual
  costs can be very small). Internally they are stored in ms.
- Changed DISK_READ_COST (was DISK_SEEK_BASE_COST) from a hard disk cost
  (9 ms) to common SSD cost (400MB/sec).
- Removed cost calculations for hard disks (rotation etc).
- Changed the following handler functions to return IO_AND_CPU_COST.
  This makes it easy to apply different cost modifiers in ha_..time()
  functions for io and cpu costs.
  - scan_time()
  - rnd_pos_time() & rnd_pos_call_time()
  - keyread_time()
- Enhanched keyread_time() to calculate the full cost of reading of a set
  of keys with a given number of ranges and optional number of blocks that
  need to be accessed.
- Removed read_time() as keyread_time() + rnd_pos_time() can do the same
  thing and more.
- Tuned cost for: heap, myisam, Aria, InnoDB, archive and MyRocks.
  Used heap table costs for json_table. The rest are using default engine
  costs.
- Added the following new optimizer variables:
  - optimizer_disk_read_ratio
  - optimizer_disk_read_cost
  - optimizer_key_lookup_cost
  - optimizer_row_lookup_cost
  - optimizer_row_next_find_cost
  - optimizer_scan_cost
- Moved all engine specific cost to OPTIMIZER_COSTS structure.
- Changed costs to use 'records_out' instead of 'records_read' when
  recalculating costs.
- Split optimizer_costs.h to optimizer_costs.h and optimizer_defaults.h.
  This allows one to change costs without having to compile a lot of
  files.
- Updated costs for filter lookup.
- Use a better cost estimate in best_extension_by_limited_search()
  for the sorting cost.
- Fixed previous issues with 'filtered' explain column as we are now
  using 'records_out' (min rows seen for table) to calculate filtering.
  This greatly simplifies the filtering code in
  JOIN_TAB::save_explain_data().

This change caused a lot of queries to be optimized differently than
before, which exposed different issues in the optimizer that needs to
be fixed.  These fixes are in the following commits.  To not have to
change the same test case over and over again, the changes in the test
cases are done in a single commit after all the critical change sets
are done.

InnoDB changes:
- Updated InnoDB to not divide big range cost with 2.
- Added cost for InnoDB (innobase_update_optimizer_costs()).
- Don't mark clustered primary key with HA_KEYREAD_ONLY. This will
  prevent that the optimizer is trying to use index-only scans on
  the clustered key.
- Disabled ha_innobase::scan_time() and ha_innobase::read_time() and
  ha_innobase::rnd_pos_time() as the default engine cost functions now
  works good for InnoDB.

Other things:
- Added  --show-query-costs (\Q) option to mysql.cc to show the query
  cost after each query (good when working with query costs).
- Extended my_getopt with GET_ADJUSTED_VALUE which allows one to adjust
  the value that user is given. This is used to change cost from
  microseconds (user input) to milliseconds (what the server is
  internally using).
- Added include/my_tracker.h  ; Useful include file to quickly test
  costs of a function.
- Use handler::set_table() in all places instead of 'table= arg'.
- Added SHOW_OPTIMIZER_COSTS to sys variables. These are input and
  shown in microseconds for the user but stored as milliseconds.
  This is to make the numbers easier to read for the user (less
  pre-zeros).  Implemented in 'Sys_var_optimizer_cost' class.
- In test_quick_select() do not use index scans if 'no_keyread' is set
  for the table. This is what we do in other places of the server.
- Added THD parameter to Unique::get_use_cost() and
  check_index_intersect_extension() and similar functions to be able
  to provide costs to called functions.
- Changed 'records' to 'rows' in optimizer_trace.
- Write more information to optimizer_trace.
- Added INDEX_BLOCK_FILL_FACTOR_MUL (4) and INDEX_BLOCK_FILL_FACTOR_DIV (3)
  to calculate usage space of keys in b-trees. (Before we used numeric
  constants).
- Removed code that assumed that b-trees has similar costs as binary
  trees. Replaced with engine calls that returns the cost.
- Added Bitmap::find_first_bit()
- Added timings to join_cache for ANALYZE table (patch by Sergei Petrunia).
- Added records_init and records_after_filter to POSITION to remember
  more of what best_access_patch() calculates.
- table_after_join_selectivity() changed to recalculate 'records_out'
  based on the new fields from best_access_patch()

Bug fixes:
- Some queries did not update last_query_cost (was 0). Fixed by moving
  setting thd->...last_query_cost in JOIN::optimize().
- Write '0' as number of rows for const tables with a matching row.

Some internals:
- Engine cost are stored in OPTIMIZER_COSTS structure.  When a
  handlerton is created, we also created a new cost variable for the
  handlerton. We also create a new variable if the user changes a
  optimizer cost for a not yet loaded handlerton either with command
  line arguments or with SET
  @@global.engine.optimizer_cost_variable=xx.
- There are 3 global OPTIMIZER_COSTS variables:
  default_optimizer_costs   The default costs + changes from the
                            command line without an engine specifier.
  heap_optimizer_costs      Heap table costs, used for temporary tables
  tmp_table_optimizer_costs The cost for the default on disk internal
                            temporary table (MyISAM or Aria)
- The engine cost for a table is stored in table_share. To speed up
  accesses the handler has a pointer to this. The cost is copied
  to the table on first access. If one wants to change the cost one
  must first update the global engine cost and then do a FLUSH TABLES.
  This was done to be able to access the costs for an open table
  without any locks.
- When a handlerton is created, the cost are updated the following way:
  See sql/keycaches.cc for details:
  - Use 'default_optimizer_costs' as a base
  - Call hton->update_optimizer_costs() to override with the engines
    default costs.
  - Override the costs that the user has specified for the engine.
  - One handler open, copy the engine cost from handlerton to TABLE_SHARE.
  - Call handler::update_optimizer_costs() to allow the engine to update
    cost for this particular table.
  - There are two costs stored in THD. These are copied to the handler
    when the table is used in a query:
    - optimizer_where_cost
    - optimizer_scan_setup_cost
- Simply code in best_access_path() by storing all cost result in a
  structure. (Idea/Suggestion by Igor)
2023-02-02 23:54:45 +03:00
Sergei Golubchik
eb26bf6e09 unify client/tool version string
it should now always be

/path/to/exe Ver <tool version> Distrib <server version> for <OS> (<ARCH>)

in all tools and clients
2023-01-19 12:39:28 +01:00
Marko Mäkelä
92c8d6f168 Merge 10.7 into 10.8
The MDEV-25004 test innodb_fts.versioning is omitted because ever since
commit 685d958e38 InnoDB would not allow
writes to a database where the redo log file ib_logfile0 is missing.
2023-01-10 14:42:50 +02:00
Marko Mäkelä
8356fb68c3 Merge 10.6 into 10.7 2023-01-04 14:52:25 +02:00
Marko Mäkelä
e441c32a0b Merge 10.5 into 10.6 2023-01-03 18:13:11 +02:00
Marko Mäkelä
8b9b4ab3f5 Merge 10.4 into 10.5 2023-01-03 17:08:42 +02:00
Marko Mäkelä
fb0808c450 Merge 10.3 into 10.4 2023-01-03 16:10:02 +02:00
musvaage
7c5609fb64 typos 2022-12-21 12:46:52 +11:00
Oleksandr Byelkin
f3fddc1b4a Merge branch '10.7' into 10.8 2022-10-17 08:44:12 +02:00
Oleksandr Byelkin
ec2b30e736 Merge branch '10.6' into 10.7 2022-10-16 21:40:33 +02:00
Oleksandr Byelkin
822694bd56 Merge branch '10.5' into 10.6 2022-10-15 23:47:33 +02:00
Marko Mäkelä
66e44afd94 Merge 10.4 into 10.5 2022-10-13 17:05:30 +03:00
Marko Mäkelä
f404911557 Merge 10.3 into 10.4 2022-10-13 16:50:26 +03:00
Marko Mäkelä
618d820646 Merge 10.7 into 10.8 2022-10-13 10:42:41 +03:00
Marko Mäkelä
588efca237 Merge 10.6 into 10.7 2022-10-13 10:05:29 +03:00
Nikita Malyavin
3cd2c1e8b6 MDEV-29299 SELECT from table with vcol index reports warning
As of now innodb does not store trx_id for each record in secondary index.
The idea behind is following: let us store only per-page max_trx_id, and
delete-mark the records when they are deleted/updated.

If the read starts, it rememders the lowest id of currently active
transaction. Innodb refers to it as trx->read_view->m_up_limit_id.
See also ReadView::open.

When the page is fetched, its max_trx_id is compared to m_up_limit_id.
If the value is lower, and the secondary index record is not delete-marked,
then this page is just safe to read as is. Else, a clustered index could be
needed ato access. See page_get_max_trx_id call in row_search_mvcc, and the
corresponding switch (row_search_idx_cond_check(...)) below.

Virtual columns are required to be updated in case if the record was
delete-marked. The motivation behind it is documented in
Row_sel_get_clust_rec_for_mysql::operator() near
row_sel_sec_rec_is_for_clust_rec call.

This was basically a description why virtual column computation can
normally happen during SELECT, and, generally, a vcol index access.

Sometimes stats tables are updated by innodb. This starts a new
transaction, and it can happen that it didn't finish to the moment of
SELECT execution, forcing virtual columns recomputation. If the result was
a something that normally outputs a warning, like division by zero, then
it could be outputted in a racy manner.

The solution is to suppress the warnings when a column is computed
for the described purpose.
ignore_wrnings argument is added innobase_get_computed_value.
Currently, it is only true for a call from
row_sel_sec_rec_is_for_clust_rec.
2022-10-12 20:49:45 +03:00
Marko Mäkelä
a992c615a6 Merge 10.5 into 10.6 2022-10-12 12:14:13 +03:00
Marko Mäkelä
977c385df3 Merge 10.4 into 10.5 2022-10-12 11:29:32 +03:00
Alexander Barkov
3416315407 A followup for MDEV-29672 Add MTR tests covering key and key segment flags and types
Adding debug output for key and keyseg flags at ha_myisam::open() time.
So now there are three points of debug output:

1. In the very end of mysql_prepare_create_table()
2. In ha_myisam::create(), after the table2myisam() call
3. In ha_myisan::open(), after the mi_open() call

mi_create(), which is is called between 2 and 3, modifies flags for
some data types, so the output in 2 and 3 is different.
2022-10-10 14:10:48 +04:00
Marko Mäkelä
6dc157f8a6 Merge 10.5 into 10.6 2022-10-06 09:22:39 +03:00
Marko Mäkelä
de078e060e Merge 10.4 into 10.5 2022-10-06 08:29:56 +03:00
Marko Mäkelä
df97eb1432 Remove HAVE_SNPRINTF
This fixes up commit 77c184df7c
which explicitly specifies that we use ISO/IEC 9899:1999 (C99),
which includes the snprintf() function.
2022-10-05 10:09:49 +03:00
Oleksandr Byelkin
2f70784c2a Merge branch '10.7' into 10.8 2022-10-04 11:42:37 +02:00
Oleksandr Byelkin
b6ebadaa66 Merge branch '10.6' into 10.7 2022-10-04 07:41:35 +02:00
Sergei Golubchik
900d7bf360 Merge branch '10.5' into 10.6 2022-10-02 22:14:21 +02:00
Sergei Golubchik
3a2116241b Merge branch '10.4' into 10.5 2022-10-02 14:38:13 +02:00
Alexander Barkov
1118e979c2 MDEV-29672 Add MTR tests covering key and key segment flags and types 2022-09-30 11:08:49 +04:00
Marko Mäkelä
829e8111c7 Merge 10.5 into 10.6 2022-09-26 14:34:43 +03:00
Marko Mäkelä
6286a05d80 Merge 10.4 into 10.5 2022-09-26 13:34:38 +03:00
Marko Mäkelä
a69cf6f07e MDEV-29613 Improve WITH_DBUG_TRACE=OFF
In commit 28325b0863
a compile-time option was introduced to disable the macros
DBUG_ENTER and DBUG_RETURN or DBUG_VOID_RETURN.

The parameter name WITH_DBUG_TRACE would hint that it also
covers DBUG_PRINT statements. Let us do that: WITH_DBUG_TRACE=OFF
shall disable DBUG_PRINT() as well.

A few InnoDB recovery tests used to check that some output from
DBUG_PRINT("ib_log", ...) is present. We can live without those checks.

Reviewed by: Vladislav Vaintroub
2022-09-23 13:40:42 +03:00
Marko Mäkelä
4345d93100 Merge 10.7 into 10.8 2022-09-21 09:52:09 +03:00
Marko Mäkelä
7c7ac6d4a4 Merge 10.6 into 10.7 2022-09-21 09:33:07 +03:00
Marko Mäkelä
44fd2c4b24 Merge 10.5 into 10.6 2022-09-20 16:53:20 +03:00
Marko Mäkelä
0792aff161 Merge 10.4 into 10.5 2022-09-20 13:17:02 +03:00
Marko Mäkelä
0c0a569028 Merge 10.3 into 10.4 2022-09-20 12:38:25 +03:00
Marko Mäkelä
5e959bc363 Fix clang -Wunused-but-set-variable 2022-09-19 13:30:52 +03:00
Marko Mäkelä
18bb95b608 Merge 10.7 into 10.8 2022-03-14 11:52:11 +02:00
Marko Mäkelä
e67d46e4a1 Merge 10.6 into 10.7 2022-03-14 11:30:32 +02:00
Marko Mäkelä
572e34304e Merge 10.5 into 10.6 2022-03-14 10:59:46 +02:00
Marko Mäkelä
c2146ce774 MDEV-24841: More workarounds
For some reason, the tests of the MemorySanitizer build on 10.5 failed
with both clang 13 and clang 14 with SIGSEGV. On 10.6 where it worked
better, some more places to work around were identified.
2022-03-14 10:37:39 +02:00
Marko Mäkelä
89cd3da48c Merge 10.7 into 10.8 2022-03-11 15:56:59 +02:00
Marko Mäkelä
3c9f415e52 Merge 10.6 into 10.7 2022-03-11 14:52:16 +02:00
Marko Mäkelä
42cb400562 Merge 10.5 into 10.6 2022-03-11 13:35:35 +02:00
Marko Mäkelä
97d82808b8 Fix clang -Wtypedef-redefinition
This fixes commit 77c184df7c.
2022-03-11 13:29:41 +02:00
Sergei Golubchik
72b5b8b2e3 MDEV-27434 DESC attribute does not work with auto-increment on secondary column of multi-part index
when searching for the last auto-inc value, it's HA_READ_PREFIX_LAST for
the ASC keypart, but HA_READ_PREFIX for the DESC one

also fixes MDEV-27585
2022-01-26 18:43:06 +01:00
Sergei Golubchik
d7e7f48eb4 MDEV-27407 Different ASC/DESC index attributes on MERGE and underlying table can cause wrong results
detect if merge children are "differently defined" regarding ASC/DESC
2022-01-26 18:43:06 +01:00
Sergei Golubchik
ddbb3d1447 MDEV-27309 Server crash or ASAN memcpy-param-overlap upon INSERT into Aria/MyISAM table with DESC key
MyiSAM and Aria, indexes with prefix compression, where the first keypart
could be NULL - in this case they didn't expect the next key after the
not NULL key to be NULL.

Expect the first keypart of the next key to have zero length even
if store_not_null==1, this combination means keypart is NULL,
don't pack it.

also fixes MDEV-27340
2022-01-26 18:43:06 +01:00
Sergei Golubchik
799a30660e cleanup: reduce code duplication 2022-01-26 18:43:06 +01:00
Sergei Golubchik
775e7ce6d6 MDEV-27303 Table corruption after insert into a non-InnoDB table with DESC index
optimized prefix search didn't take into account descending indexes

also fixes MDEV-27330
2022-01-26 18:43:06 +01:00