Commit graph

25218 commits

Author SHA1 Message Date
Aleksey Midenkov
d324c03d0c Vanilla cleanups and refactorings
Dead code cleanup:

part_info->num_parts usage was wrong and working incorrectly in
mysql_drop_partitions() because num_parts is already updated in
prep_alter_part_table(). We don't have to update part_info->partitions
because part_info is destroyed at alter_partition_lock_handling().

Cleanups:

- DBUG_EVALUATE_IF() macro replaced by shorter form DBUG_IF();
- Typo in ER_KEY_COLUMN_DOES_NOT_EXITS.

Refactorings:

- Splitted write_log_replace_delete_frm() into write_log_delete_frm()
  and write_log_replace_frm();
- partition_info via DDL_LOG_STATE;
- set_part_info_exec_log_entry() removed.

DBUG_EVALUATE removed

DBUG_EVALUTATE was only added for consistency together with
DBUG_EVALUATE_IF. It is not used anywhere in the code.

DBUG_SUICIDE() fix on release build

On release DBUG_SUICIDE() was statement. It was wrong as
DBUG_SUICIDE() is used in expression context.
2021-10-26 17:07:46 +02:00
Thirunarayanan Balathandayuthapani
045757af4c MDEV-24621 In bulk insert, pre-sort and build indexes one page at a time
When inserting a number of rows into an empty table,
InnoDB will buffer and pre-sort the records for each index, and
build the indexes one page at a time.

For each index, a buffer of innodb_sort_buffer_size will be created.

If the buffer ran out of memory then we will create temporary files
for storing the data.

At the end of the statement, we will sort and apply the buffered
records. Ideally, we would do this at the end of the transaction
or only when starting to execute a non-INSERT statement on the table.
However, it could be awkward if duplicate keys or similar errors
would be reported during the execution of a later statement.
This will be addressed in MDEV-25036.

Any columns longer than 2000 bytes will buffered in temporary files.

innodb_prepare_commit_versioned(): Apply all bulk buffered insert
operation, at the end of each statement.

ha_commit_trans(): Handle errors from innodb_prepare_commit_versioned().

row_merge_buf_write(): This function should accept blob
file handle too and it should write the field data which are
greater than 2000 bytes

row_merge_bulk_t: Data structure to maintain the data during
bulk insert operation.

trx_mod_table_time_t::start_bulk_insert(): Notify the start of
bulk insert operation and create new buffer for the given table

trx_mod_table_time_t::add_tuple(): Buffer a record.

trx_mod_table_time_t::write_bulk(): Do bulk insert operation
present in the transaction

trx_mod_table_time_t::bulk_buffer_exist(): Whether the buffer
storage exist for the bulk transaction

trx_mod_table_time_t::write_bulk(): Write all buffered insert
operation for the transaction and the table.

row_ins_clust_index_entry_low(): Insert the data into the
bulk buffer if it is already exist.

row_ins_sec_index_entry(): Insert the secondary tuple
if the bulk buffer already exist.

row_merge_bulk_buf_add(): Insert the tuple into bulk buffer
insert operation.

row_merge_buf_blob(): Write the field data whose length is
more than 2000 bytes into blob temporary file. Write the
file offset and length into the tuple field.

row_merge_copy_blob_from_file(): Copy the blob from blob file
handler based on reference of the given tuple.

row_merge_insert_index_tuples(): Handle blob for bulk insert
operation.

row_merge_bulk_t::row_merge_bulk_t(): Constructor. Initialize
the buffer and file for all the indexes expect fts index.

row_merge_bulk_t::create_tmp_file(): Create new temporary file
for the given index.

row_merge_bulk_t::write_to_tmp_file(): Write the content from
buffer to disk file for the given index.

row_merge_bulk_t::add_tuple(): Insert the tuple into the merge
buffer for the given index. If the memory ran out then InnoDB
should sort the buffer and write into file.

row_merge_bulk_t::write_to_index(): Do bulk insert operation
from merge file/merge buffer for the given index

row_merge_bulk_t::write_to_table(): Do bulk insert operation
for all the indexes.

dict_stats_update(): If a bulk insert transaction is in progress,
treat the table as empty. The index creation could hold latches
for extended amounts of time.
2021-10-26 15:01:44 +03:00
Marko Mäkelä
c8e309a6ac Merge 10.6 into 10.7 2021-10-26 15:01:22 +03:00
Marko Mäkelä
58fe6b47d4 MDEV-26903: Assertion ctx->trx->state == TRX_STATE_ACTIVE on DROP INDEX
rollback_inplace_alter_table(): Tolerate a case where the transaction
is not in an active state. If ha_innobase::commit_inplace_alter_table()
failed with a deadlock, the transaction would already have been
rolled back. This omission of error handling was introduced in
commit 1bd681c8b3 (MDEV-25506 part 3).

After commit c3c53926c4 (MDEV-26554)
it became easier to trigger DB_DEADLOCK during exclusive table lock
acquisition in ha_innobase::commit_inplace_alter_table().

lock_table_low(): Add DBUG injection "innodb_table_deadlock".
2021-10-26 09:54:37 +03:00
Vladislav Vaintroub
30009f2999 Fix 32bit build 2021-10-25 10:46:20 +02:00
Marko Mäkelä
1193a793c4 MDEV-26674: Set innodb_use_native_aio=OFF when using io_uring on a potentially affected kernel
We have observed hangs of the io_uring subsystem when using a
Linux kernel newer than 5.10. Also 5.15-rc6 is affected by this.

The exact cause of the hangs has not been diagnosed yet.
As a safety measure, we will disable innodb_use_native_aio by default
when the server has been configured with io_uring and the kernel
version is between 5.11 and 5.15.

If the start-up parameter innodb_use_native_aio=ON is set, then
we will issue a warning to the server error log.
2021-10-25 10:55:04 +03:00
Marko Mäkelä
6bfaa68c62 MDEV-26882 InnoDB number of trx pools note improvement 2021-10-22 14:42:00 +03:00
Marko Mäkelä
71d4ecf182 Merge 10.6 into 10.7 2021-10-22 14:41:47 +03:00
Marko Mäkelä
1f02280904 MDEV-26769 InnoDB does not support hardware lock elision
This implements memory transaction support for:

* Intel Restricted Transactional Memory (RTM), also known as TSX-NI
(Transactional Synchronization Extensions New Instructions)
* POWER v2.09 Hardware Trace Monitor (HTM) on GNU/Linux

transactional_lock_guard, transactional_shared_lock_guard:
RAII lock guards that try to elide the lock acquisition
when transactional memory is available.

buf_pool.page_hash: Try to elide latches whenever feasible.
Related to the InnoDB change buffer and ROW_FORMAT=COMPRESSED
tables, this is not always possible.
In buf_page_get_low(), memory transactions only work reasonably
well for validating a guessed block address.

TMLockGuard, TMLockTrxGuard, TMLockMutexGuard: RAII lock guards
that try to elide lock_sys.latch and related latches.
2021-10-22 12:38:45 +03:00
Marko Mäkelä
c091a0bc8d MDEV-26826 Duplicated computations of buf_pool.page_hash addresses
Since commit bd5a6403ca (MDEV-26033)
we can actually calculate the buf_pool.page_hash cell and latch
addresses while not holding buf_pool.mutex.

buf_page_alloc_descriptor(): Remove the MEM_UNDEFINED.
We now expect buf_page_t::hash to be zero-initialized.

buf_pool_t::hash_chain: Dedicated data type for buf_pool.page_hash.array.

buf_LRU_free_one_page(): Merged to the only caller
buf_pool_t::corrupted_evict().
2021-10-22 12:33:37 +03:00
Marko Mäkelä
fdae71f8e3 MDEV-26828 Spinning on buf_pool.page_hash is wasting CPU cycles
page_hash_latch: Only use the spinlock implementation on
SUX_LOCK_GENERIC platforms (those for which we do not implement
a futex-like interface). Use srw_spin_mutex on 32-bit systems
(except Microsoft Windows) to satisfy the size constraints.

rw_lock::is_read_locked(): Remove. We will use the slightly
broader assertion is_locked().

srw_lock_: Implement is_locked(), is_write_locked() in a hacky
way for the Microsoft Windows SRWLOCK. This should be acceptable,
because we are only using these predicates in debug assertions
(or later, in lock elision), and false positives should not matter.
2021-10-22 12:32:26 +03:00
Marko Mäkelä
5caff20216 MDEV-26883 InnoDB hang due to table lock conflict
In a stress test campaign of a 10.6-based branch by Matthias Leich,
a deadlock between two InnoDB threads occurred, involving
lock_sys.wait_mutex and a dict_table_t::lock_mutex.

The cause of the hang is a latching order violation in
lock_sys_t::cancel(). That function and the latching order
violation were originally introduced in
commit 8d16da1487 (MDEV-24789).

lock_sys_t::cancel(): Invoke table->lock_mutex_trylock() in order
to avoid a deadlock. If that fails, release lock_sys.wait_mutex,
and acquire both latches. In that way, we will be obeying the
latching order and no hangs will occur.

This hang should mostly affect DDL operations. DML operations will
acquire only IX or IS table locks, which are compatible with each other.
2021-10-22 11:59:44 +03:00
Marko Mäkelä
73f5cbd0b6 Merge 10.5 into 10.6 2021-10-21 16:06:34 +03:00
Marko Mäkelä
a0fda162eb Fix GCC 11.2.0 -m32 (IA-32) warnings
page_create_low(): Fix -Warray-bounds

log_buffer_extend(): Fix -Wstringop-overflow
2021-10-21 15:31:21 +03:00
Marko Mäkelä
5f8561a6bc Merge 10.4 into 10.5 2021-10-21 15:26:25 +03:00
Marko Mäkelä
489ef007be Merge 10.3 into 10.4 2021-10-21 14:57:00 +03:00
Marko Mäkelä
d5bcccdabb Merge 10.2 into 10.3 2021-10-21 14:38:07 +03:00
Marko Mäkelä
fbb1e92e25 MDEV-19522 fixup: Integer type mismatch in unit test 2021-10-21 14:35:23 +03:00
Marko Mäkelä
e4a7c15dd6 Merge 10.2 into 10.3 2021-10-21 13:41:04 +03:00
Marko Mäkelä
1a2308d3f4 MDEV-26865: Add test case and instrumentation
Based on mysql/mysql-server@bc9c46bf28
but without sleeps.

The test was verified to hit the debug assertion if the change to
fts_add_doc_by_id() in commit 2d98b967e3
was reverted.
2021-10-21 12:57:09 +03:00
Marko Mäkelä
2d98b967e3 MDEV-26865 fts_optimize_thread cannot keep up with workload
fts_cache_t::total_size_at_sync: New field, to sample total_size.

fts_add_doc_by_id(): Invoke sync if total_size has grown too much
since the previous sync request. (Maintain cache->total_size_at_sync.)

ib_wqueue_t::length: Caches ib_list_len(*items).

ib_wqueue_len(): Removed. We will refer to fts_optimize_wq->length
directly.

Based on mysql/mysql-server@bc9c46bf28
2021-10-21 12:56:59 +03:00
Marko Mäkelä
c484a358c8 MDEV-26864 Race condition between transaction commit and undo log truncation
trx_commit_in_memory(): Do not release the rseg reference before
trx_undo_commit_cleanup() has been invoked and the current transaction
is truly done with the rollback segment. The purpose of the reference
count is to prevent data races with trx_purge_truncate_history().

This is based on
mysql/mysql-server@ac79aa1522.
2021-10-21 12:56:59 +03:00
Thirunarayanan Balathandayuthapani
8ce8c269f4 MDEV-19522 InnoDB commit fails when FTS_DOC_ID value is greater than 4294967295
InnoDB commit fails when consecutive FTS_DOC_ID value
is greater than 4294967295.
Fix is that InnoDB should remove the delta FTS_DOC_ID
value limitations and fts should encode 8 byte value,
remove FTS_DOC_ID_MAX_STEP variable. Replaced the
fts0vlc.ic file with fts0vlc.h

fts_encode_int(): Should be able to encode 10 bytes value

fts_get_encoded_len(): Should get the length of the value
which has 10 bytes

fts_decode_vlc(): Add debug assertion to verify the maximum
length allowed is 10.

mach_read_uint64_little_endian(): Reads 64 bit stored in
little endian format

Added a unit test case which check for minimum and maximum
value to do the fts encoding
2021-10-21 12:56:59 +03:00
Marko Mäkelä
6b4fad9402 MDEV-22627 fixup: Add a type cast for 32-bit platforms 2021-10-21 12:56:59 +03:00
Marko Mäkelä
05c3dced86 MDEV-22627 fixup: Cover also ALTER TABLE...ALGORITHM=INPLACE 2021-10-20 22:16:23 +03:00
Marko Mäkelä
b06e8167a7 MDEV-22627 Failing assertion: dict_tf2_is_valid(flags, flags2)
create_table_info_t::innobase_table_flags(): Refuse to create
a PAGE_COMPRESSED table with PAGE_COMPRESSION_LEVEL=0 if also
innodb_compression_level=0.

The parameter value innodb_compression_level=0 was only somewhat
meaningful for testing or debugging ROW_FORMAT=COMPRESSED tables.
For the page_compressed format, it never made any sense, and the
check in dict_tf_is_valid_not_redundant() that was added in
72378a2583 (MDEV-12873) would cause
the server to crash.
2021-10-20 16:04:29 +03:00
Marko Mäkelä
d6a3f425ee After-merge fix: Remove unused variable 2021-10-19 20:38:07 +03:00
Marko Mäkelä
6e390a62ba MDEV-26772 InnoDB DDL fails with DUPLICATE KEY error
ha_innobase::delete_table(): When the table that is being dropped
has a name starting with #sql, temporarily set
innodb_lock_wait_timeout=0 while attempting to lock the
persistent statistics tables. If the statistics tables cannot be locked,
pretend that statistics did not exist and carry on with dropping
the table. The SQL layer is not really prepared for failures of
this operation. This is what fixes the test case.

ha_innobase::rename_table(): When renaming a table from a name
that starts with #sql, try to lock the statistics tables with an
immediate timeout, and ignore the statistics if the locks were
not available. In fact, during any rename from a #sql name,
dict_stats_rename_table() should have no effect, because already
when an earlier rename to a #sql name took place we should have
deleted the statistics for the table using the non-#sql name.
This change is just analogous to the ha_innobase::delete_table().
2021-10-19 19:54:29 +03:00
Vicențiu Ciorbaru
1388845e04 Fix Groonga crash on MIPS: Correctly link to libatomic
MIPS (and possibly other) platforms require linking against libatomic to
support 64-bit atomic integers. Groonga was failing to do so and all related
tests were failing with an atomics relocation error on MIPS.

Contributors:
James Cowgill <jcowgill@debian.org>
2021-10-19 19:22:54 +03:00
Sergei Golubchik
abbc8821dc columnstore compilation fixes 2021-10-19 17:35:06 +02:00
Sergei Golubchik
26c1311c39 compilation failures on Windows 2021-10-19 17:35:06 +02:00
Vicențiu Ciorbaru
a33c1082da Fix MIPS build failure: Handle unaligned buffers in connect's TYPBLK class
On MIPS platforms (and probably others) unaligned memory access results in a
bus error. In the connect storage engine, block data for some data formats is
stored packed in memory and the TYPBLK class is used to read values from it.
Since TYPBLK does not have special handling for this packed memory, it can
quite easily result in unaligned memory accesses.

The simple way to fix this is to perform all accesses to the main buffer
through memcpy. With GCC and optimizations turned on, this call to memcpy is
completely optimized away on architectures where unaligned accesses are ok
(like x86).

Contributors:
James Cowgill <jcowgill@debian.org>
2021-10-19 16:08:51 +03:00
Eric Herman
401ff6994d MDEV-26221: DYNAMIC_ARRAY use size_t for sizes
https://jira.mariadb.org/browse/MDEV-26221
my_sys DYNAMIC_ARRAY and DYNAMIC_STRING inconsistancy

The DYNAMIC_STRING uses size_t for sizes, but DYNAMIC_ARRAY used uint.
This patch adjusts DYNAMIC_ARRAY to use size_t like DYNAMIC_STRING.

As the MY_DIR member number_of_files is copied from a DYNAMIC_ARRAY,
this is changed to be size_t.

As MY_TMPDIR members 'cur' and 'max' are copied from a DYNAMIC_ARRAY,
these are also changed to be size_t.

The lists of plugins and stored procedures use DYNAMIC_ARRAY,
but their APIs assume a size of 'uint'; these are unchanged.
2021-10-19 16:00:26 +03:00
Nayuta Yanagisawa
e7208bd934 MDEV-26158 SIGSEGV in spider_free_mem from ha_spider::open on INSERT
The server crashes due to passing NULL to spider_free().

In some cases, this == pt_handler_share_handlers[0] at the label
error_get_share in ha_spider::open().

In such cases, to nullify pt_handler_share_handlers[0]->wide_handler
is nothing but to nullify this->wide_handler. We should not do this
before freeing this->wide_handler.
2021-10-19 19:04:05 +09:00
Krunal Bauskar
f7684f0ca5 MDEV-26855: Enable spinning for log_sys_mutex and log_flush_order_mutex
As part of MDEV-26779 we first discovered the effect of enabling spinning for
some critical mutex. MDEV-26779 tried enabling it for lock_sys.wait_mutex and
observed a good gain in performance.

In yet another discussion, Mark Callaghan pointed a reference to pthread based
mutex spin using PTHREAD_MUTEX_ADAPTIVE_NP (MDEV-26769 Intel RTM).

Given the strong references, Marko Makela as part of his comment in 
pointed an idea to enable spinning for other mutexes. Based on perf profiling
we decided to explore spinning for log_sys_mutex and log_flush_order_mutex as
they are occupying the top slots in the contented mutex list.

The evaluation showed promising results for ARM64 but not for x86.
So a patch is here-by proposed to enable the spinning of the mutex for
ARM64-based platform.
2021-10-19 09:28:18 +03:00
Marko Mäkelä
c3c53926c4 MDEV-26554: Races between INSERT on child and DDL on parent table
The SQL layer never acquires metadata locks (MDL) on the tables
that the tables that DML statement accesses is modifying.

However, the storage engine must access the parent table in order to
ensure that the child table will not refer to a non-existing record
in the parent table.

During certain DDL operations, the InnoDB table metadata (dict_table_t)
may be be freed and rebuilt. This would cause a race condition with
a concurrent INSERT that is attempting to report a FOREIGN KEY violation.

We work around the insufficient MDL during DML by acquiring exclusive
InnoDB table locks on all child tables during DDL. To avoid deadlocks,
we will follow the following order of acquisition:

1. tables whose REFERENCES clauses point to the current table
2. the current table that is being subjected to DDL
3. mysql.innodb_table_stats
4. mysql.innodb_index_stats
5. the InnoDB dictionary tables (SYS_TABLES and so on)
6. exclusive dict_sys.latch
2021-10-18 18:03:12 +03:00
Marko Mäkelä
59fe6a8a01 Merge 10.5 into 10.6 2021-10-18 17:47:31 +03:00
Nayuta Yanagisawa
edde9084c2 MDEV-26582 SIGSEGV in spider_db_bulk_insert and spider_db_connect and spider_db_before_query, and hang in "End of update loop" / "Reset for next command" query states
Spider accesses a freed connection in ha_spider::end_bulk_insert()
and results in SIGSEGV.

The cause of the bug is that ha_spider::is_bulk_insert_exec_period()
wrongly returns TRUE when the bulk insertion has not yet started.

Spider decides whether it is during the bulk insertion or not by
the value of insert_pos, but the variable is not reset in a case,
and this result in the bug.
2021-10-18 23:18:42 +09:00
Marko Mäkelä
9c5835e067 Merge 10.5 into 10.6 2021-10-18 16:36:24 +03:00
Marko Mäkelä
18eab4a832 MDEV-26682 Replication timeouts with XA PREPARE
The purpose of non-exclusive locks in a transaction is to guarantee
that the records covered by those locks must remain in that way until
the transaction is committed. (The purpose of gap locks is to ensure
that a record that was nonexistent will remain that way.)

Once a transaction has reached the XA PREPARE state, the only allowed
further actions are XA ROLLBACK or XA COMMIT. Therefore, it can be
argued that only the exclusive locks that the XA PREPARE transaction
is holding are essential.

Furthermore, InnoDB never preserved explicit locks across server restart.
For XA PREPARE transations, we will only recover implicit exclusive locks
for records that had been modified.

Because of the fact that XA PREPARE followed by a server restart will
cause some locks to be lost, we might as well always release all
non-exclusive locks during the execution of an XA PREPARE statement.

lock_release_on_prepare(): Release non-exclusive locks on XA PREPARE.

trx_prepare(): Invoke lock_release_on_prepare() unless the
isolation level is SERIALIZABLE or this is an internal distributed
transaction with the binlog (not actual XA PREPARE statement).

This has been discussed with Sergei Golubchik and Andrei Elkin.

Reviewed by: Sergei Golubchik
2021-10-18 12:49:10 +03:00
Nayuta Yanagisawa
9068020efe MDEV-26539 SIGSEGV in spider_check_and_set_trx_isolation and I_P_List_iterator from THD::drop_temporary_table (10.5.3 opt only) on ALTER
The server crashes if ALTER TABLE, which accesses physical data
placed at data nodes, is performed on a Spider table.

The cause of the bug is that spider_check_trx_and_get_conn() does
not allocate connections if sql_command == SQLCOM_ALTER_TABLE.
Some ALTER TABLE statements, like ALTER TABLE ... CHECK PARTITION,
access data nodes. So, we need to allocate a new connection before
performing ALTER TABLEs.
2021-10-18 13:23:48 +09:00
Nayuta Yanagisawa
39f6315612 MDEV-19866 follow-up
Cherry-picking the fix for MDEV-19866 changes the behavior of
the Spider slightly. So, I modified a existing test to match
the new behavior.
2021-10-18 13:19:03 +09:00
Kentoku SHIBA
a46665090b MDEV-19866 With a Spider table, a SELECT with WHERE involving primary key breaks following SELECTs ()
Change checking scanning partitions from part_spec to part_info->read_partitions
2021-10-18 13:19:03 +09:00
Marko Mäkelä
b4911f5a34 Merge 10.6 into 10.7 2021-10-13 16:37:12 +03:00
Vladislav Vaintroub
78e023c274 Workaround a assertion on shutdown.
Something initiates purge, while all purge THDs are destroyed.
tpool::waitable_task.disable() would not allow the task to be executed anymore
2021-10-13 15:02:57 +02:00
Marko Mäkelä
607de9c7ac Merge 10.5 into 10.6 2021-10-13 15:17:20 +03:00
Marko Mäkelä
f1acd9f14b MDEV-26819 SET GLOBAL innodb_max_dirty_pages_pct=0 occasionally fails to trigger writes
innodb_max_dirty_pages_pct_update(),
innodb_max_dirty_pages_pct_lwm_update():
Invoke buf_pool.page_cleaner_wakeup() in order to wake up
buf_flush_page_cleaner. This allows the test innodb.page_cleaner
to run without any occasional timeouts.

The occasional hangs were introduced by
commit 7b1252c03d (MDEV-24278).
2021-10-13 15:16:23 +03:00
Marko Mäkelä
a8379e53e8 Merge 10.5 into 10.6
The changes to galera.galear_var_replicate_myisam_on
in commit d9b933bec6
are omitted due to conflicts
with commit 27d66d644c.
2021-10-13 13:28:12 +03:00
Marko Mäkelä
99bb3fb656 Merge 10.4 into 10.5 2021-10-13 12:33:56 +03:00
Marko Mäkelä
a736a3174a Merge 10.3 into 10.4 2021-10-13 12:03:32 +03:00