Commit graph

528 commits

Author SHA1 Message Date
sjaakola
7bffe468b2 MDEV-21910 Deadlock between BF abort and manual KILL command
When high priority replication slave applier encounters lock conflict in innodb,
it will force the conflicting lock holder transaction (victim) to rollback.
This is a must in multi-master sychronous replication model to avoid cluster lock-up.
This high priority victim abort (aka "brute force" (BF) abort), is started
from innodb lock manager while holding the victim's transaction's (trx) mutex.
Depending on the execution state of the victim transaction, it may happen that the
BF abort will call for THD::awake() to wake up the victim transaction for the rollback.
Now, if BF abort requires THD::awake() to be called, then the applier thread executed
locking protocol of: victim trx mutex -> victim THD::LOCK_thd_data

If, at the same time another DBMS super user issues KILL command to abort the same victim,
it will execute locking protocol of: victim THD::LOCK_thd_data  -> victim trx mutex.
These two locking protocol acquire mutexes in opposite order, hence unresolvable mutex locking
deadlock may occur.

The fix in this commit adds THD::wsrep_aborter flag to synchronize who can kill the victim
This flag is set both when BF is called for from innodb and by KILL command.
Either path of victim killing will bail out if victim's wsrep_killed is already
set to avoid mutex conflicts with the other aborter execution. THD::wsrep_aborter
records the aborter THD's ID. This is needed to preserve the right to kill
the victim from different locations for the same aborter thread.
It is also good error logging, to see who is reponsible for the abort.

A new test case was added in galera.galera_bf_kill_debug.test for scenario where
wsrep applier thread and manual KILL command try to kill same idle victim
2020-07-22 08:20:10 +03:00
Vladislav Vaintroub
98333883ee MDEV-22933 - remove ---source include/not_threadpool.inc from tests 2020-06-18 23:12:54 +02:00
Marko Mäkelä
5155a300fa MDEV-22871: Reduce InnoDB buf_pool.page_hash contention
The rw_lock_s_lock() calls for the buf_pool.page_hash became a
clear bottleneck after MDEV-15053 reduced the contention on
buf_pool.mutex. We will replace that use of rw_lock_t with a
special implementation that is optimized for memory bus traffic.

The hash_table_locks instrumentation will be removed.

buf_pool_t::page_hash: Use a special implementation whose API is
compatible with hash_table_t, and store the custom rw-locks
directly in buf_pool.page_hash.array, intentionally sharing
cache lines with the hash table pointers.

rw_lock: A low-level rw-lock implementation based on std::atomic<uint32_t>
where read_trylock() becomes a simple fetch_add(1).

buf_pool_t::page_hash_latch: The special of rw_lock for the page_hash.

buf_pool_t::page_hash_latch::read_lock(): Assert that buf_pool.mutex
is not being held by the caller.

buf_pool_t::page_hash_latch::write_lock() may be called while not holding
buf_pool.mutex. buf_pool_t::watch_set() is such a caller.

buf_pool_t::page_hash_latch::read_lock_wait(),
page_hash_latch::write_lock_wait(): The spin loops.
These will obey the global parameters innodb_sync_spin_loops and
innodb_sync_spin_wait_delay.

buf_pool_t::freed_page_hash: A singly linked list of copies of
buf_pool.page_hash that ever existed. The fact that we never
free any buf_pool.page_hash.array guarantees that all
page_hash_latch that ever existed will remain valid until shutdown.

buf_pool_t::resize_hash(): Replaces buf_pool_resize_hash().
Prepend a shallow copy of the old page_hash to freed_page_hash.

buf_pool_t::page_hash_table::n_cells: Declare as Atomic_relaxed.

buf_pool_t::page_hash_table::lock(): Explain what prevents a
race condition with buf_pool_t::resize_hash().
2020-06-18 14:16:01 +03:00
Marko Mäkelä
5579c38991 MDEV-22884: Adjust the test for PLUGIN_PERFSCHEMA=NO 2020-06-14 18:58:55 +03:00
Sergei Golubchik
9ed08f3576 MDEV-22884 Assertion `grant_table || grant_table_role' failed on perfschema
when allowing access via perfschema callbacks, update
the cached GRANT_INFO to match
2020-06-13 21:22:07 +02:00
Monty
4102f1589c Aria will now register it's transactions
MDEV-22531 Remove maria::implicit_commit()
MDEV-22607 Assertion `ha_info->ht() != binlog_hton' failed in
           MYSQL_BIN_LOG::unlog_xa_prepare

From the handler point of view, Aria now looks like a transactional
engine. One effect of this is that we don't need to call
maria::implicit_commit() anymore.

This change also forces the server to call trans_commit_stmt() after doing
any read or writes to system tables.  This work will also make it easier
to later allow users to have system tables in other engines than Aria.

To handle the case that Aria doesn't support rollback, a new
handlerton flag, HTON_NO_ROLLBACK, was added to engines that has
transactions without rollback (for the moment only binlog and Aria).

Other things
- Moved freeing of MARIA_SHARE to a separate function as the MARIA_SHARE
  can be still part of a transaction even if the table has closed.
- Changed Aria checkpoint to use the new MARIA_SHARE free function. This
  fixes a possible memory leak when using S3 tables
- Changed testing of binlog_hton to instead test for HTON_NO_ROLLBACK
- Removed checking of has_transaction_manager() in handler.cc as we can
  assume that as the transaction was started by the engine, it does
  support transactions.
- Added new class 'start_new_trans' that can be used to start indepdendent
  sub transactions, for example while reading mysql.proc, using help or
  status tables etc.
- open_system_tables...() and open_proc_table_for_Read() doesn't anymore
  take a Open_tables_backup list. This is now handled by 'start_new_trans'.
- Split thd::has_transactions() to thd::has_transactions() and
  thd::has_transactions_and_rollback()
- Added handlerton code to free cached transactions objects.
  Needed by InnoDB.

squash! 2ed35999f2a2d84f1c786a21ade5db716b6f1bbc
2020-05-23 12:29:10 +03:00
Sergei Golubchik
13038e4705 Merge branch '10.4' into 10.5 2020-05-09 20:43:36 +02:00
Oleksandr Byelkin
0253ea7f22 MDEV-19650: Privilege bug on MariaDB 10.4
Also fixes:
MDEV-21487: Implement option for mysql_upgrade that allows root@localhost to be replaced
MDEV-21486: Implement option for mysql_install_db that allows root@localhost to be replaced

Add user mariadb.sys to be definer of user view
(and has right on underlying table global_priv for
required operation over global_priv
(SELECT,UPDATE,DELETE))

Also changed definer of gis functions in case of creation,
but they work with any definer so upgrade script do not try
to push this change.
2020-05-07 10:54:56 +02:00
Monty
78357796e8 Added support for VISIBLE attribute for indexes in CREATE TABLE
MDEV-22199 Add VISIBLE attribute for indexes in CREATE TABLE

This was done to make it easier to read in dumps from MySQL 8.0 generated
with MySQL workbench
2020-04-19 17:33:51 +03:00
Rasmus Johansson
9075973dbf MDEV-17812 Use MariaDB in error messages instead of MySQL
Changed wording in error messages from MySQL to MariaDB. In
cases where the word server could be used instead it was done.

Tests that have these errors recorded were updated.
2020-04-08 06:09:42 +00:00
Nikita Malyavin
259fb1cbed MDEV-16978 Application-time periods: WITHOUT OVERLAPS
* The overlaps check is implemented on a handler level per row command.
  It creates a separate cursor (actually, another handler instance) and
  caches it inside the original handler, when ha_update_row or
  ha_insert_row is issued. Cursor closes on unlocking the handler.

* Containing the same key in index means unique constraint violation
  even in usual terms. So we fetch left and right neighbours and check
  that they have same key prefix, excluding from the key only the period part.
  If it doesnt match, then there's no such neighbour, and the check passes.
  Otherwise, we check if this neighbour intersects with the considered key.

* The check does not introduce new error and fails with ER_DUPP_KEY error.
  This might break REPLACE workflow and should be fixed separately
2020-03-31 17:42:34 +02:00
Monty
6736f152f4 Added FLUSH THREADS 2020-03-24 21:00:04 +02:00
Sergei Golubchik
02fe997505 fix a nondeterminism in perfschema.statement_program_non_nested test
when selecting from perfschema, filter out statements
used by the test istself in wait_condition.inc, because they,
by design, can be repeated unpredictable number of times.
2020-03-23 10:45:08 +01:00
Alexander Barkov
e0eacbee77 MDEV-21975 Add BINLOG REPLAY privilege and bind new privileges to gtid_seq_no, preudo_thread_id, server_id, gtid_domain_id 2020-03-18 20:16:34 +04:00
Sergei Golubchik
b55d960e43 fix a nondeterminism in perfschema.statement_program_nesting_event_check test
when selecting from perfschema, filter out statements
used by the test istself in wait_condition.inc, because they,
by design, can be repeated unpredictable number of times.
2020-03-16 10:33:38 +01:00
Sergei Golubchik
c4f3e37b0c fix a race condition in the perfschema.transaction_nested_events 2020-03-14 09:51:52 +01:00
Sergei Golubchik
df25d67d5f perfschema test formatting. Use --echo # 2020-03-14 09:51:36 +01:00
Marko Mäkelä
4f4fccecb2 Fix perfschema.statement_program_concurrency 2020-03-11 08:29:48 +02:00
Alexander Barkov
0e04beb28f Recoding new suite/perfschema/r/start_server_low_digest_sql_length.result
Fixing a test failure introduced by MDEV-21743
2020-03-11 00:35:30 +04:00
Alexander Barkov
a1e330de5a MDEV-21743 Split up SUPER privilege to smaller privileges 2020-03-10 23:49:47 +04:00
Sergei Golubchik
7180afa094 fix perfschema for pool-of-threads 2020-03-10 19:24:24 +01:00
Sergei Golubchik
7af733a5a2 perfschema compilation, test and misc fixes 2020-03-10 19:24:23 +01:00
Sergei Golubchik
0ea717f51a P_S 5.7.28 2020-03-10 19:24:22 +01:00
Oleksandr Byelkin
4b087e1754 Merge branch '10.4' into 10.5 2020-02-12 08:55:17 +01:00
Oleksandr Byelkin
646d1ec83a Merge branch '10.3' into 10.4 2020-02-11 14:40:35 +01:00
Oleksandr Byelkin
fafb35ee51 MDEV-20076: SHOW GRANTS does not quote role names properly
Quotes added to output.
2020-02-05 17:22:26 +01:00
Marko Mäkelä
a983b24407 Merge 10.4 into 10.5 2020-01-28 14:17:09 +02:00
Oleksandr Byelkin
2833e90619 fix perfschema.start_server_innodb test (related to MDEV-17571) 2020-01-24 14:34:07 +01:00
Oleksandr Byelkin
6918157e98 Merge branch '10.3' into 10.4 2020-01-21 23:15:02 +01:00
Oleksandr Byelkin
ade89fc898 Merge branch '10.2' into 10.3 2020-01-21 09:11:14 +01:00
Oleksandr Byelkin
3a1716a7e7 Merge branch '10.1' into 10.2 2020-01-20 16:15:05 +01:00
Marko Mäkelä
ded128aa9b Merge 10.4 into 10.5 2020-01-20 16:48:56 +02:00
Marko Mäkelä
87a61355e8 Merge 10.3 into 10.4
The MDEV-17062 fix in commit c4195305b2
was omitted.
2020-01-20 15:49:48 +02:00
Oleksandr Byelkin
10eacd5ff7 Merge branch 'merge-perfschema-5.6' into 10.1 2020-01-19 13:11:45 +01:00
Oleksandr Byelkin
3aff3f3679 5.6.47 2020-01-19 12:52:07 +01:00
Sergei Petrunia
e709eb9bf7 Merge branch '10.2' into 10.3
# Conflicts:
#	mysql-test/suite/galera/r/MW-388.result
#	mysql-test/suite/galera/t/MW-388.test
#	mysql-test/suite/innodb/r/truncate_inject.result
#	mysql-test/suite/innodb/t/truncate_inject.test
#	mysql-test/suite/rpl/r/rpl_stop_slave.result
#	mysql-test/suite/rpl/t/rpl_stop_slave.test
#	sql/sp_head.cc
#	sql/sp_head.h
#	sql/sql_lex.cc
#	sql/sql_yacc.yy
#	storage/xtradb/buf/buf0dblwr.cc
2020-01-17 00:46:40 +03:00
Alice Sherepa
b7fb30e930 MDEV-21360 global debug_dbug pre-test value restoration issues 2020-01-15 18:06:24 +01:00
Alice Sherepa
451573fab1 MDEV-21360 debug_dbug pre-test value restoration issues 2020-01-15 18:06:24 +01:00
Vladislav Vaintroub
5e62b6a5e0 MDEV-16264 Use threadpool for Innodb background work.
Almost all threads have gone
- the "ticking" threads, that sleep a while then do some work)
(srv_monitor_thread, srv_error_monitor_thread, srv_master_thread)
were replaced with timers. Some timers are periodic,
e.g the "master" timer.

- The btr_defragment_thread is also replaced by a timer , which
reschedules it self when current defragment "item" needs throttling

- the buf_resize_thread and buf_dump_threads are substitutes with tasks
Ditto with page cleaner workers.

- purge workers threads are not tasks as well, and purge cleaner
coordinator is a combination of a task and timer.

- All AIO is outsourced to tpool, Innodb just calls thread_pool::submit_io()
and provides the callback.

- The srv_slot_t was removed, and innodb_debug_sync used in purge
is currently not working, and needs reimplementation.
2019-11-15 18:09:30 +01:00
Marko Mäkelä
77e8a311e1 Merge 10.4 into 10.5
A conflict between MDEV-19514 (b42294bc64)
and MDEV-20934 (d7a2401750)
was resolved. We will not invoke the function ibuf_delete_recs()
from ibuf_merge_or_delete_for_page(). Instead, we will add that
logic to the function ibuf_read_merge_pages().
2019-11-07 10:34:33 +02:00
Marko Mäkelä
928abd6967 Merge 10.3 into 10.4 2019-11-06 13:44:56 +02:00
Marko Mäkelä
908ca4668d Merge 10.2 into 10.3 2019-11-06 13:14:31 +02:00
Marko Mäkelä
8688ef22c2 Merge 10.1 to 10.2 2019-11-06 10:18:51 +02:00
Robert Bindar
6f86150ab3 MDEV-17896 Assertion `pfs->get_refcount() > 0' failed
Unfortunate DROP TEMPORARY..IF EXISTS on a regular table may allow
subsequent CREATE TABLE statements to steal away the PFS_table_share
instance from the dropped table.
2019-11-01 11:10:04 +02:00
Marko Mäkelä
4081b7b27a Merge 10.4 into 10.5 2019-09-06 17:16:40 +03:00
Sergei Golubchik
244f0e6dd8 Merge branch '10.3' into 10.4 2019-09-06 11:53:10 +02:00
Monty
a071e0e029 Merge branch '10.2' into 10.3 2019-09-03 13:17:32 +03:00
Monty
9cba6c5aa3 Updated mtr files to support different compiled in options
This allows one to run the test suite even if any of the following
options are changed:
- character-set-server
- collation-server
- join-cache-level
- log-basename
- max-allowed-packet
- optimizer-switch
- query-cache-size and query-cache-type
- skip-name-resolve
- table-definition-cache
- table-open-cache
- Some innodb options
etc

Changes:
- Don't print out the value of system variables as one can't depend on
  them to being constants.
- Don't set global variables to 'default' as the default may not
  be the same as the test was started with if there was an additional
  option file. Instead save original value and reset it at end of test.
- Test that depends on the latin1 character set should include
  default_charset.inc or set the character set to latin1
- Test that depends on the original optimizer switch, should include
  default_optimizer_switch.inc
- Test that depends on the value of a specific system variable should
  set it in the test (like optimizer_use_condition_selectivity)
- Split subselect3.test into subselect3.test and subselect3.inc to
  make it easier to set and reset system variables.
- Added .opt files for test that required specfic options that could
  be changed by external configuration files.
- Fixed result files in rockdsb & tokudb that had not been updated for
  a while.
2019-09-01 19:17:35 +03:00
Marko Mäkelä
826f9d4f7e Merge 10.4 into 10.5 2019-05-23 10:32:21 +03:00
Sergey Vojtovich
a61baa7a25 Maintain connection_count atomically
Removed LOCK_connection_count.

Removed duplicate denied_connections and connection_errors_max_connection
increment from create_new_thread(). Another increment done by
CONNECT::close_with_error().

Simplified away CONNECT::thread_count_incremented. Now close_with_error()
is always called with connection_count incremented.

Part of MDEV-19515 - Improve connect speed
2019-05-21 17:55:09 +04:00