Commit graph

1773 commits

Author SHA1 Message Date
Monty
2347c18c79 Renamed maria_* source files for executables to aria_* 2020-05-27 15:47:10 +03:00
Monty
c52e62a76f Improve logging of Aria redo's and undo's
- Remove extra ',' and quotes
- Remove extra newline and remove double newlines
- Added options --lsn-redo-end and --lsn-undo-end to aria_read_log
- Allow one to give the aria_read_log lsn aruments as number,0xhexnumber,
  the same way as lsn's are written by aria_read_log
- Don't write full pages to redo log with EXTRA_DEBUG as this takes up
  a lot of disk and there has not been a need for this extra loggging for
  a long time. Instead one should use EXTRA_ARIA_DEBUG instead.
2020-05-27 15:47:10 +03:00
Monty
514533eb9a aria_chk sets --warning-if-wrong-transid=0 if --ignore-control-file is used 2020-05-27 15:47:10 +03:00
Monty
59f011943f MDEV-22686 # Assertion `trn' failed in ha_maria::start_stmt
Calling stmt rollback closed the transaction, which caused the above issue
2020-05-24 19:29:56 +03:00
Monty
c4bf4b7aef Fixed access to undefined memory found by valgrind and MSAN
When my_vsnprintf() is patched, the code protected disabled with
'WAITING_FOR_BUGFIX_TO_VSPRINTF' should be enabled again. Also all %b
formats in this patch should be revert to %s again
2020-05-23 12:29:10 +03:00
Monty
4ea171ffab Fixed compiler warnings from gcc and clang 5.0.1 2020-05-23 12:29:10 +03:00
Monty
4102f1589c Aria will now register it's transactions
MDEV-22531 Remove maria::implicit_commit()
MDEV-22607 Assertion `ha_info->ht() != binlog_hton' failed in
           MYSQL_BIN_LOG::unlog_xa_prepare

From the handler point of view, Aria now looks like a transactional
engine. One effect of this is that we don't need to call
maria::implicit_commit() anymore.

This change also forces the server to call trans_commit_stmt() after doing
any read or writes to system tables.  This work will also make it easier
to later allow users to have system tables in other engines than Aria.

To handle the case that Aria doesn't support rollback, a new
handlerton flag, HTON_NO_ROLLBACK, was added to engines that has
transactions without rollback (for the moment only binlog and Aria).

Other things
- Moved freeing of MARIA_SHARE to a separate function as the MARIA_SHARE
  can be still part of a transaction even if the table has closed.
- Changed Aria checkpoint to use the new MARIA_SHARE free function. This
  fixes a possible memory leak when using S3 tables
- Changed testing of binlog_hton to instead test for HTON_NO_ROLLBACK
- Removed checking of has_transaction_manager() in handler.cc as we can
  assume that as the transaction was started by the engine, it does
  support transactions.
- Added new class 'start_new_trans' that can be used to start indepdendent
  sub transactions, for example while reading mysql.proc, using help or
  status tables etc.
- open_system_tables...() and open_proc_table_for_Read() doesn't anymore
  take a Open_tables_backup list. This is now handled by 'start_new_trans'.
- Split thd::has_transactions() to thd::has_transactions() and
  thd::has_transactions_and_rollback()
- Added handlerton code to free cached transactions objects.
  Needed by InnoDB.

squash! 2ed35999f2a2d84f1c786a21ade5db716b6f1bbc
2020-05-23 12:29:10 +03:00
Monty
d1d472646d Change THD->transaction to a pointer to enable multiple transactions
All changes (except one) is of type
thd->transaction.  -> thd->transaction->

thd->transaction points by default to 'thd->default_transaction'
This allows us to 'easily' have multiple active transactions for a
THD object, like when reading data from the mysql.proc table
2020-05-23 12:29:10 +03:00
Monty
7ae812cf2c Fix that BACKUP STAGE BLOCK_COMMIT blocks commit to the Aria engine
MDEV-22468 BACKUP STAGE BLOCK_COMMIT should block commit in the Aria engine

This is needed to ensure that mariabackup works properly with Aria tables

This code ads new calls to ha_maria::implicit_commit(). These will be
deleted by MDEV-22531 Remove maria::implicit_commit().
2020-05-23 12:29:10 +03:00
Monty
b15615631f MDEV-18286 Assertion `pagecache->cnt_for_resize_op ...
This patch is also pushed in 10.4. It's pushed separately in 10.5 as there
are some additional cases in 10.5 to take care of.

When merging if there is conflicts, use this code, not the 10.4 code.
2020-05-23 12:29:10 +03:00
Monty
7cb160961c Changed debug label XXX in ma_pagecache.cc XXX to proper lables 2020-05-23 12:29:10 +03:00
Monty
c287dc9b04 Fixed typo in variable description 2020-05-23 12:29:09 +03:00
Monty
8cf166bfb2 Update libmarias3 to fix a memory leak 2020-05-23 12:29:09 +03:00
Marko Mäkelä
5ece2155cb Merge 10.4 into 10.5 2020-05-20 17:46:05 +03:00
Marko Mäkelä
2bf93a8fd6 Merge 10.3 into 10.4 2020-05-19 21:18:15 +03:00
Marko Mäkelä
79ed33c184 Merge 10.2 into 10.3 2020-05-19 17:05:05 +03:00
Monty
141cf43e61 Fixed assert in Aria on SHOW PROCEDURE STATUS
MDEV-18457 Assertion `(bitmap->map +
           (bitmap->full_head_size/6*6)) <= full_head_end failed

The problem was that full_head_size was not calculated correctly
in the case when insert_order was inforced, which is the case
for SHOW commands.
2020-05-19 03:42:54 +03:00
Sergei Golubchik
13038e4705 Merge branch '10.4' into 10.5 2020-05-09 20:43:36 +02:00
Sergei Petrunia
8d85715d50 MDEV-21794: Optimizer flag rowid_filter leads to long query
Rowid Filter check is just like Index Condition Pushdown check: before
we check the filter, we must check if we have walked out of the range
we are scanning. (If we did, we should return, and not continue the scan).

Consequences of this:
- Rowid filtering doesn't work for keys that have partially-covered
  blob columns (just like Index Condition Pushdown)
- The rowid filter function has three return values: CHECK_POS (passed)
  CHECK_NEG (filtered out), CHECK_OUT_OF_RANGE.

All of the above is implemented in this patch
2020-05-07 12:27:17 +02:00
Marko Mäkelä
496d0372ef Merge 10.4 into 10.5 2020-04-29 15:40:51 +03:00
Marko Mäkelä
b63446984c Merge 10.3 into 10.4 2020-04-27 17:38:17 +03:00
Marko Mäkelä
2e12d471ea Merge 10.2 into 10.3 2020-04-27 14:24:41 +03:00
Marko Mäkelä
c06845d6f0 Merge 10.1 into 10.2 2020-04-27 13:28:13 +03:00
Marko Mäkelä
758fbec6e3 Fix clang 10 warnings
_ma_fetch_keypage(): Correct an assertion that used to always hold.
Thanks to clang -Wint-in-bool-context for flagging this.

double_to_datetime_with_warn(): Suppress -Wimplicit-int-float-conversion
by adding a cast. LONGLONG_MAX converted to double will actually be
LONGLONG_MAX+1.
2020-04-27 09:40:34 +03:00
Marko Mäkelä
6a3fc1101c Aria: Avoid unused variables in embedded server 2020-04-27 09:40:10 +03:00
Monty
81511b4776 Fixed shutdown crash in Aria that affects debug binaries
MDEV-18286 Assertion `pagecache->cnt_for_resize_op == 0' failed in
check_pagecache_is_cleaned_up on server shutdown

The reason for the crash is that the counter-of-pinned-pages in the
Aria pagecache goes wrong.
This only affects debug builds, as in these we do an assert on shutdown
if the counter-of-pinned-pages is not 0 (some page was left pinned).

The bug was that in 2 places in the page cache, when not succeeding to
pin a page and a retry was made, the counter-of-pinned-pages counter was
not properly adjusted.

In the given test case, BLOCK_COMMIT flushed all Aria files. If a block was flushed at the same time the insert tried to access it, the insert would retry to get the block and that would cause the counter to go wrong.
2020-04-26 15:26:48 +03:00
Marko Mäkelä
fbe2712705 Merge 10.4 into 10.5
The functional changes of commit 5836191c8f
(MDEV-21168) are omitted due to MDEV-742 having addressed the issue.
2020-04-25 21:57:52 +03:00
Monty
9398c3dfa5 Fixed memory leak if Aria didn't start
This is not crtitical as if Aria doesn't start, the server will exit at once.
2020-04-23 14:24:52 +03:00
Marko Mäkelä
88cf6f1c7f Merge 10.3 into 10.4 2020-04-22 18:18:51 +03:00
Marko Mäkelä
455cf6196c Merge 10.2 into 10.3 2020-04-22 14:45:55 +03:00
Marko Mäkelä
e52a36d37b Merge 10.1 into 10.2 2020-04-22 13:50:53 +03:00
Monty
8d74d30dde Fixed compiler warning 2020-04-19 17:33:51 +03:00
Monty
f40ca33bbc Make all #sql temporary table names uniform
The reason for this is to make all temporary file names similar and
also to be able to figure out from where a #sql-xxx name orginates.

New format is for most cases:
'#sql-name-current_pid-thread_id[-increment]'
Where name is one of subselect, alter, exchange, temptable or backup

The exceptions are:
ALTER PARTITION shadow files:
'#sql-shadow-thread_id-'original_table_name'

Names used with temp pool:
'#sql-name-current_pid-pool_number'
2020-04-19 17:33:51 +03:00
Monty
eca5c2c67f Added support for more functions when using partitioned S3 tables
MDEV-22088 S3 partitioning support

All ALTER PARTITION commands should now work on S3 tables except

REBUILD PARTITION
TRUNCATE PARTITION
REORGANIZE PARTITION

In addition, PARTIONED S3 TABLES can also be replicated.
This is achived by storing the partition tables .frm and .par file on S3
for partitioned shared (S3) tables.

The discovery methods are enchanced by allowing engines that supports
discovery to also support of the partitioned tables .frm and .par file

Things in more detail

- The .frm and .par files of partitioned tables are stored in S3 and kept
  in sync.
- Added hton callback create_partitioning_metadata to inform handler
  that metadata for a partitoned file has changed
- Added back handler::discover_check_version() to be able to check if
  a table's or a part table's definition has changed.
- Added handler::check_if_updates_are_ignored(). Needed for partitioning.
- Renamed rebind() -> rebind_psi(), as it was before.
- Changed CHF_xxx hadnler flags to an enum
- Changed some checks from using table->file->ht to use
  table->file->partition_ht() to get discovery to work with partitioning.
- If TABLE_SHARE::init_from_binary_frm_image() fails, ensure that we
  don't leave any .frm or .par files around.
- Fixed that writefrm() doesn't leave unusable .frm files around
- Appended extension to path for writefrm() to be able to reuse to function
  for creating .par files.
- Added DBUG_PUSH("") to a a few functions that caused a lot of not
  critical tracing.
2020-04-19 17:33:51 +03:00
Monty
a6d32976ae Fixed memory leak with fulltext indexes
MDEV-22275 Assertion `global_status_var.global_memory_used == 0'
failed, bytes lost, or LeakSanitizer: detected memory leaks
after using temporary table with fulltext key

This affected MyISAM and Aria temporary tables
2020-04-18 11:51:42 +03:00
Marko Mäkelä
ad4b70562b Fix GCC 10 -Woverflow
maria_page_crc_check_index(): Do not attempt to convert
HA_ERR_WRONG_CRC (176) to my_bool (char).
On platforms where char is signed, the 176 will be converted to -80.

It turns out that the callers only care whether the result is zero.
Let us return 1 in this case, like we do in all other error cases.
2020-04-17 12:02:22 +03:00
Sergey Vojtovich
5679a2b6b3 Shrink my_atomic.h and my_cpu.h scope 2020-04-15 22:23:03 +04:00
Sergei Golubchik
fcd84da5f1 MDEV-22218 InnoDB: Failing assertion: node->pcur->rel_pos == BTR_PCUR_ON upon LOAD DATA with NO_BACKSLASH_ESCAPES in SQL_MODE and unique blob in table
`inited == NONE` at the initialization time does not always mean
that it'll be `NONE` later, at the execution time. Use a more complex
caller-specific logic to decide whether to create a cloned lookup handler.

Besides LOAD (as in the original bug report) make sure that all
prepare_for_insert() invocations are covered by tests. Add tests for
CREATE ... SELECT, multi-UPDATE, and multi-DELETE.

Don't enable write cache with long uniques.
2020-04-12 22:10:57 +02:00
Daniel Black
e8351934b6
Merge pull request from grooverdan/10.4-MDEV-18851-multiple-sized-large-page-support
MDEV-18851: multiple sized large page support (linux)
2020-04-02 23:54:08 +04:00
Marko Mäkelä
6be56dd1c8 MDEV-20377: Enable MemorySanitizer user-poisoning
For ENGINE=Aria, we work around bugs in various tests that catch
writes of uninitialized bytes from the Aria page cache.
(Why do we even write anything on DROP TABLE?)
2020-03-28 21:42:59 +02:00
Andrew Hutchings
0181384a3f MDEV-20329 Fix S3 engine OpenSSL race
With OpenSSL < 1.1 there is a potential for a race condition to occur.
This can cause the S3 engine to crash. The workaround is to add locking
callbacks to OpenSSL so that this doesn't happen.

https://curl.haxx.se/libcurl/c/threadsafe.html

There is a fix in libMariaS3 for this which when a certain flag is set
(HAVE_CURL_OPENSSL_UNSAFE) will add the required locks.

This patch adds CMake support so that the flag is set if it is found
that Curl is compiled with an unsafe OpenSSL version. For example Ubuntu
16.04 with libcurl4-openssl-dev.
2020-03-27 13:02:02 +02:00
Monty
eb483c5181 Updated optimizer costs in multi_range_read_info_const() and sql_select.cc
- multi_range_read_info_const now uses the new records_in_range interface
- Added handler::avg_io_cost()
- Don't calculate avg_io_cost() in get_sweep_read_cost if avg_io_cost is
  not 1.0.  In this case we trust the avg_io_cost() from the handler.
- Changed test_quick_select to use TIME_FOR_COMPARE instead of
  TIME_FOR_COMPARE_IDX to align this with the rest of the code.
- Fixed bug when using test_if_cheaper_ordering where we didn't use
  keyread if index was changed
- Fixed a bug where we didn't use index only read when using order-by-index
- Added keyread_time() to HEAP.
  The default keyread_time() was optimized for blocks and not suitable for
  HEAP. The effect was the HEAP prefered table scans over ranges for btree
  indexes.
- Fixed get_sweep_read_cost() for HEAP tables
- Ensure that range and ref have same cost for simple ranges
  Added a small cost (MULTI_RANGE_READ_SETUP_COST) to ranges to ensure
  we favior ref for range for simple queries.
- Fixed that matching_candidates_in_table() uses same number of records
  as the rest of the optimizer
- Added avg_io_cost() to JT_EQ_REF cost. This helps calculate the cost for
  HEAP and temporary tables better. A few tests changed because of this.
- heap::read_time() and heap::keyread_time() adjusted to not add +1.
  This was to ensure that handler::keyread_time() doesn't give
  higher cost for heap tables than for normal tables. One effect of
  this is that heap and derived tables stored in heap will prefer
  key access as this is now regarded as cheap.
- Changed cost for index read in sql_select.cc to match
  multi_range_read_info_const(). All index cost calculation is now
  done trough one function.
- 'ref' will now use quick_cost for keys if it exists. This is done
  so that for '=' ranges, 'ref' is prefered over 'range'.
- scan_time() now takes avg_io_costs() into account
- get_delayed_table_estimates() uses block_size and avg_io_cost()
- Removed default argument to test_if_order_by_key(); simplifies code
2020-03-27 03:58:32 +02:00
Monty
f36ca142f7 Added page_range to records_in_range() to improve range statistics
Prototype change:
-  virtual ha_rows records_in_range(uint inx, key_range *min_key,
-                                   key_range *max_key)
+  virtual ha_rows records_in_range(uint inx, const key_range *min_key,
+                                   const key_range *max_key,
+                                   page_range *res)

The handler can ignore the page_range parameter. In the case the handler
updates the parameter, the optimizer can deduce the following:
- If previous range's last key is on the same block as next range's first
  key
- If the current key range is in one block
- We can also assume that the first and last block read are cached!
  This can be used for a better calculation of IO seeks when we
  estimate the cost of a range index scan.

The parameter is fully implemented for MyISAM, Aria and InnoDB.
A separate patch will update handler::multi_range_read_info_const() to
take the benefits of this change and also remove the double
records_in_range() calls that are not anymore needed.
2020-03-27 03:54:45 +02:00
Monty
bb8df68521 Disable building S3 for embedded server 2020-03-25 16:30:53 +02:00
Monty
3faa15cfbd Don't give warnings from safemalloc for aria_pack --help 2020-03-24 21:00:05 +02:00
Monty
37393bea23 Replace handler::primary_key_is_clustered() with handler::pk_is_clustering_key()
This was done to both simplify the code and also to be easier to handle
storage engines that are clustered on some other index than the primary
key.

As pk_is_clustering_key() and is_clustering_key now are using only
index_flags, these where removed from all storage engines.
2020-03-24 21:00:04 +02:00
Monty
8eba777c2b mysqld --help will now load mysqld.options table
Changes:
- Initalize Aria early to allow it to load mysql.plugin table with --help
- Don't print 'aborting' when doing --help
- Don't write 'loose' error messages on log_warning < 2 (2 is default)
- Don't write warnings about disabled plugings when doing --help
- Don't write aria_log_control or aria log files when doing --help
- When using --help, open all Aria tables in read only mode (safety)
- If aria_init() fails, do a cleanup(). (Frees used memory)
- If aria_log_control is locked with --help, then don't wait 30 seconds
  but instead return at once without initialzing Aria plugin.
2020-03-24 21:00:04 +02:00
Monty
44790f58ac Fixed typo in aria_s3_copy 2020-03-24 21:00:03 +02:00
Monty
4ef437558a Improve update handler (long unique keys on blobs)
MDEV-21606 Improve update handler (long unique keys on blobs)
MDEV-21470 MyISAM and Aria start_bulk_insert doesn't work with long unique
MDEV-21606 Bug fix for previous version of this code
MDEV-21819 2 Assertion `inited == NONE || update_handler != this'

- Move update_handler from TABLE to handler
- Move out initialization of update handler from ha_write_row() to
  prepare_for_insert()
- Fixed that INSERT DELAYED works with update handler
- Give an error if using long unique with an autoincrement column
- Added handler function to check if table has long unique hash indexes
- Disable write cache in MyISAM and Aria when using update_handler as
  if cache is used, the row will not be inserted until end of statement
  and update_handler would not find conflicting rows.
- Removed not used handler argument from
  check_duplicate_long_entries_update()
- Syntax cleanups
  - Indentation fixes
  - Don't use single character indentifiers for arguments
2020-03-24 21:00:02 +02:00
Monty
736998cb75 Cleanups & indentation changes
- Only indentation changes in sql_rename.cc
- Ignore some WSREP error messages when there isn't a internet connection
- Force restart of stat_tables_part.test to make result stable
- Fixed compiler warnings in CONNECT
2020-03-24 21:00:02 +02:00