Commit graph

25872 commits

Author SHA1 Message Date
Marko Mäkelä
3dff84cd15 Merge 10.6 into 10.7 2022-07-01 17:45:29 +03:00
Marko Mäkelä
62a20f8047 Merge 10.5 into 10.6 2022-07-01 15:24:50 +03:00
Marko Mäkelä
f09687094c Merge 10.4 into 10.5 2022-07-01 14:42:02 +03:00
Marko Mäkelä
392ee571c1 Merge 10.3 into 10.4 2022-07-01 13:10:36 +03:00
Thirunarayanan Balathandayuthapani
990cde800a MDEV-28912 NON-UNIQUE FTS_DOC_ID index mistaken as FTS_DOC_ID_INDEX
- InnoDB mistakenly identifies the non-unique FTS_DOC_ID index as
FTS_DOC_ID_INDEX while loading the table. dict_load_indexes()
should check whether the index is unique before assigning
fts_doc_id_index
2022-07-01 13:03:52 +03:00
Marko Mäkelä
7c35ad16e3 MDEV-28389 fixup: Fix pre-GCC 10 -Wconversion
Before version 10, GCC would think that a right shift of an
unsigned char returns int. Let us explicitly cast that back,
to silence a bogus -Wconversion warning.
2022-07-01 13:02:43 +03:00
Marko Mäkelä
045771c050 Fix most clang-15 -Wunused-but-set-variable
Also, refactor trx_i_s_common_fill_table() to remove dead code.

Warnings about yynerrs in Bison-generated yyparse() will remain for now.
2022-07-01 09:48:36 +03:00
Marko Mäkelä
90792e4a43 Merge 10.5 into 10.6 2022-06-30 19:41:07 +03:00
Thirunarayanan Balathandayuthapani
99de8cc028 MDEV-28919 Assertion `(((core_null) + 7) >> 3) == oindex.n_core_null_bytes || !not_redundant()' failed
- In case of discarded tablespace, InnoDB can't read the root page to
assign the n_core_null_bytes. Consecutive instant DDL fails because
of non-matching n_core_null_bytes.
2022-06-30 17:30:13 +05:30
Marko Mäkelä
a1267724cb MDEV-26293 InnoDB: Failing assertion: space->is_ready_to_close() ...
fil_space_t::acquire_low(): Introduce a parameter that specifies
which flags should be avoided. At all times, referenced() must not
be incremented if the STOPPING flag is set. When fil_system.mutex
is not being held by the current thread, the reference must not be
incremented if the CLOSING flag is set (unless NEEDS_FSYNC is set,
in fil_space_t::flush()).

fil_space_t::acquire(): Invoke acquire_low(STOPPING | CLOSING).
In this way, the reference count cannot be incremented after
fil_space_t::try_to_close() invoked fil_space_t::set_closing().

If the CLOSING flag was set, we must retry acquire_low() after
acquiring fil_system.mutex.

fil_space_t::prepare_acquired(): Replaces prepare(true).

fil_space_t::acquire_and_prepare(): Replaces prepare().
This basically retries fil_space_t::acquire() after
acquiring fil_system.mutex.
2022-06-30 14:28:16 +03:00
Thirunarayanan Balathandayuthapani
eb7e24932b MDEV-28806 Assertion `flag == 1' failure in row_build_index_entry_low upon concurrent ALTER and UPDATE
- During online ADD INDEX, InnoDB was incorrectly writing log for an
UPDATE that does not affect the being-created index.
2022-06-30 15:28:24 +05:30
Thirunarayanan Balathandayuthapani
e34f878139 MDEV-28706 Redundant InnoDB table fails during alter
- Redundant InnoDB table fails to set the flags2 while loading
the table. It leads to "Upgrade index name failure" during alter
operation. InnoDB should set the flags2 to FTS_AUX_HEX_NAME when
fts is being loaded
2022-06-30 11:19:49 +05:30
Marko Mäkelä
cac6f0a8c4 Merge 10.6 into 10.7 2022-06-29 16:17:14 +03:00
Marko Mäkelä
c1e3fc0e0d MDEV-28977: mariabackup.huge_lsn,strict_full_crc32 fails in 10.8
recv_sys_t::recover_deferred(): Hold the exclusive page latch until
the tablespace has been set up. Otherwise, the write of the page
may be lost due to non-existent tablespace. This race only affects
the recovery of the first page in a newly created tablespace.

This race condition was introduced in MDEV-24626.
2022-06-29 15:48:44 +03:00
Marko Mäkelä
ac0af4ec4a Merge 10.6 into 10.7 2022-06-28 08:34:12 +03:00
Nayuta Yanagisawa
dbd562787a MDEV-24343 Spider Left join failed Unknown column 't0.ID' in 'on clause'
The Spider mixes the comma join with other join types, and thus
ERROR 1054 occurs. This is well-known issue caused by the higher
precedence of JOIN over the comma (,).

We can fix the problem simply by using JOINs instead of commas.
2022-06-28 05:23:14 +09:00
Nayuta Yanagisawa
5375f0b495 MDEV-21310 AUTO_INCREMENT column throws range error on INSERT in partitioned table | Assertion `part_share->auto_inc_initialized || !can_use_for_auto_inc_init()' failed.
The bug is caused by a similar mechanism as MDEV-21027.

The function, check_insert_or_replace_autoincrement, failed to open
all the partitions on INSERT SELECT statements and it results in the
assertion error.
2022-06-28 01:48:55 +09:00
Nayuta Yanagisawa
a26700cca5 MDEV-28352 Spider: heap-use-after-free in ha_spider::lock_tables(), heap freed by spider_commit()
The heap-use-after-free is caused by the following mechanism:

  * In the execution of FLUSH TABLE WITH READ LOCK, the function
    spider_free_trx_conn() is called and the connections held by
    SPIDER_TRX::trx_conn_hash are freed.

  * Then, an instance of ha_spider maintains the freed connections
    because they are also referenced from ha_spider::conns.
    The ha_spider instance is kept in a lock structure until the
    corresponding table is unlocked.

  * Spider accesses ha_spider::conns on the implicit UNLOCK TABLE
    issued by BEGIN.

In the first place, when the connections have been freed, it means
that there are really no remote table locked by Spider.
Thus, there is no need for Spider to access ha_spider::cons on the
implicit UNLOCK TABLE.

We can fix the bug by removing the above mentioned access to
ha_spider::conns. We also modified spider_free_trx_conn() so that it
frees the connections only when no table is locked to reduce the
chance of another heap-use-after-free on ha_spider::conns.
2022-06-28 01:03:33 +09:00
Nayuta Yanagisawa
c4bfb61803 MDEV-28479 fixup: Update spider/bg and spider/handler suites 2022-06-27 22:57:33 +09:00
Marko Mäkelä
1ae8160710 MDEV-26979 heap-use-after-free or SIGSEGV when accessing INNODB_SYS_TABLESTATS during DDL
i_s_dict_fill_sys_tablestats(): Read all fields of dict_table_t
while holding dict_sys.latch.

dict_sys_t::allow_eviction(): Remove.
2022-06-27 16:51:53 +03:00
Marko Mäkelä
20cf63fe8b Merge 10.5 into 10.6 2022-06-27 16:51:27 +03:00
Marko Mäkelä
773f1dad94 Merge 10.4 into 10.5 2022-06-27 16:17:02 +03:00
Marko Mäkelä
b922ae5fc9 Merge 10.3 into 10.4 2022-06-27 16:16:20 +03:00
Marko Mäkelä
f339ef3f97 MDEV-26577 InnoDB: Failing assertion: dict_tf2_is_valid(flags, flags2) during ADD COLUMN
prepare_inplace_alter_table_dict(): If the table will not be rebuilt,
preserve all of the original ROW_FORMAT, including the compressed
page size flags related to ROW_FORMAT=COMPRESSED.
2022-06-27 16:00:34 +03:00
Marko Mäkelä
a75ad73545 MDEV-28389 fixup: Fix compiler warnings
hex_to_ascii(): Add #if around the definition to avoid
clang -Wunused-function. Avoid GCC 5 -Wconversion with a cast.
2022-06-27 14:50:00 +03:00
Marko Mäkelä
39f45f6f89 MDEV-28950 Assertion `*err == DB_SUCCESS' failed in btr_page_split_and_insert
btr_root_raise_and_insert(), btr_lift_page_up(),
rtr_page_split_and_insert(): Reset DB_FAIL from a failure to
copy records on a ROW_FORMAT=COMPRESSED page to DB_SUCCESS
before retrying.

This fixes a regression that was introduced by
commit 0b47c126e3 (MDEV-13542).

btr_root_raise_and_insert(): Remove a redundant condition.
btr_page_split_and_insert() will invoke btr_page_split_and_insert()
if needed.
2022-06-27 12:32:03 +03:00
Marko Mäkelä
87bd79b1e7 Merge 10.5 into 10.6 2022-06-27 10:59:31 +03:00
Marko Mäkelä
ea847cbeaf Merge 10.4 into 10.5 2022-06-27 10:51:20 +03:00
Marko Mäkelä
03174cabd7 Fix GCC -Og -Wmaybe-uninitialized 2022-06-27 10:49:15 +03:00
Marko Mäkelä
dd7e9fb38a MDEV-28854 after-merge fix: Remove a test for MDEV-26583 2022-06-27 10:47:05 +03:00
Marko Mäkelä
01d757036f Merge 10.3 into 10.4 2022-06-27 10:14:37 +03:00
Marko Mäkelä
c86b1389de MDEV-28389: Simplify the InnoDB corrupted page output
buf_page_print(): Dump the buffer page 32 bytes (64 hexadecimal digits)
per line. In this way, the limitation in mtr
("Data too long for column 'line'") will not be triggered.

Also, do not bother decoding the page contents, because everything
is present in the hexadecimal output.

dict_index_find_on_id_low(): Merge to dict_index_get_if_in_cache_low().
The direct call in buf_page_print() was prone to crashing, in case the
table definition was concurrently evicted or dropped from the
data dictionary cache.
2022-06-27 09:49:49 +03:00
Hirokazu Hata
2c1aaa6664
MDEV-28854 Disallow INSERT DELAYED on Spider table
Spider supports (or at least allows) INSERT DELAYED but the
documentation does not specify spider as a storage engine that supports
"INSERT DELAYED".
Also, although not mentioned in the documentation, "INSERT DELAYED" is
not intended to be executed inside a transaction, as can be seen from
the list of supported storage engines.
The current implementation allows executing a delayed insert on a
remote transactional table and this breaks the consistency ensured by
the transaction.

We too remove "internal_delayed", one of the Spider table parameters.
Documentation says,

> Whether to transmit existence of delay to remote servers when
> executing an INSERT DELAYED statement on local server.

This table parameter is only used for "INSERT DELAYED".

Reviewed by: Nayuta Yanagisawa
2022-06-27 14:58:18 +09:00
Nayuta Yanagisawa
925999bb97 MDEV-28829 Deprecate spider_semi_table_lock and spider_semi_table_lock_connection
When the variable, spider_semi_table_lock, is 1, Spider sends
LOCK TABLES before each SQL execution. The feature is for
non-transactional remote tables and adds some overhead to query
executions.

We change the default value of the plugin variable to 0 and then
deprecate the variable because it is rare to use non-transactional
engines these days and the variable complicates the code.

The variable, spider_semi_table_lock_connection, should be too
deprecated because it is for tweaking the semi-table locking.
2022-06-27 11:14:10 +09:00
Nayuta Yanagisawa
8e6c8967a1 MDEV-28479 Deprecate Spider's high availability feature
Deprecate the high availability feature of Spider as the feature is
beyond the scope of the federated storage engine and it complicates
the implementation of Spider.
2022-06-27 11:11:32 +09:00
Vladislav Vaintroub
d96436c999 MDEV-28935 crash in io_slots::release
Revert "TSAN: data race on vptr (ctor/dtor vs virtual call)"

This reverts commit 78084fa747.

This commit was done to please TSAN, which falsely reported an error
where there was none.
Yet as consequence, it could cause a real error, a crash in os_aio_free on
shutdown
2022-06-23 17:42:32 +02:00
Marko Mäkelä
5d0496c749 Merge 10.6 into 10.7 2022-06-23 13:20:25 +03:00
Marko Mäkelä
f2f18e20eb MDEV-28923 atomic.rename_table occasionally fails
fil_name_process(): If the recovery of a tablespace was deferred,
do invoke fil_ibd_load() even though the name in recv_spaces is
not changing. This allows us to recover from a situation where
there are many FILE_RENAME records, renaming a tablespace back
and forth, and a FILE_MODIFY record that had been written by
fil_names_clear().

Co-developed with: Thirunarayanan Balathandayuthapani
2022-06-23 13:17:20 +03:00
Marko Mäkelä
6f4d0659dd MDEV-22388 Corrupted undo log record leads to server crash
trx_undo_rec_copy(): Return nullptr if the undo record is corrupted.

trx_undo_rec_get_undo_no(): Define inline with the declaration.

trx_purge_dummy_rec: Replaced with a -1 pointer.

row_undo_rec_get(), UndorecApplier::apply_undo_rec(): Check
if trx_undo_rec_copy() returned nullptr.

trx_purge_get_next_rec(): Return nullptr upon encountering any
corruption, to signal the end of purge.
2022-06-22 10:04:28 +03:00
Marko Mäkelä
8ebff3bcb0 Merge 10.6 into 10.7 2022-06-22 08:34:29 +03:00
Marko Mäkelä
0fa19fdebf MDEV-28836 fixup
On GNU/Linux, even though the C11 aligned_alloc() appeared in
GNU libc early on, some custom memory allocators did not
implement it until recently. For example, before
gperftools/gperftools@d406f22853
the free() in tcmalloc would fail to free memory that was
returned by aligned_alloc(), because the latter would map to the
built-in allocator of libc. The Linux specific memalign() has a
similar interface and is safer to use, because it has been
available for a longer time. For AddressSanitizer, we will use
aligned_alloc() so that the constraint on size can be enforced.

buf_tmp_reserve_compression_buf(): When HAVE_ALIGNED_ALLOC holds,
round up the size to be an integer multiple of the alignment.

pfs_malloc(): In the unit test stub, round up the size to be an
integer multiple of the alignment.
2022-06-22 08:23:32 +03:00
Marko Mäkelä
6680fd8d4b Merge 10.6 into 10.7 2022-06-21 18:02:41 +03:00
Marko Mäkelä
3794673111 MDEV-28836: Memory alignment cleanup
Table_cache_instance: Define the structure aligned at
the CPU cache line, and remove a pad[] data member.
Krunal Bauskar reported this to improve performance on ARMv8.

aligned_malloc(): Wrapper for the Microsoft _aligned_malloc()
and the ISO/IEC 9899:2011 <stdlib.h> aligned_alloc().
Note: The parameters are in the Microsoft order (size, alignment),
opposite of aligned_alloc(alignment, size).
Note: The standard defines that size must be an integer multiple
of alignment. It is enforced by AddressSanitizer but not by GNU libc
on Linux.

aligned_free(): Wrapper for the Microsoft _aligned_free() and
the standard free().

HAVE_ALIGNED_ALLOC: A new test. Unfortunately, support for
aligned_alloc() may still be missing on some platforms.
We will fall back to posix_memalign() for those cases.

HAVE_MEMALIGN: Remove, along with any use of the nonstandard memalign().

PFS_ALIGNEMENT (sic): Removed; we will use CPU_LEVEL1_DCACHE_LINESIZE.

PFS_ALIGNED: Defined using the C++11 keyword alignas.

buf_pool_t::page_hash_table::create(),
lock_sys_t::hash_table::create():
lock_sys_t::hash_table::resize(): Pad the allocation size to an
integer multiple of the alignment.

Reviewed by: Vladislav Vaintroub
2022-06-21 16:59:49 +03:00
Marko Mäkelä
2e43af69e3 MDEV-28870 InnoDB: Missing FILE_CREATE, FILE_DELETE or FILE_MODIFY before FILE_CHECKPOINT
There was a race condition between log_checkpoint_low() and
deleting or renaming data files. The scenario is as follows:

1. The buffer pool does not contain dirty pages.
2. A FILE_DELETE or FILE_RENAME record is written.
3. The checkpoint LSN will be moved ahead of the write of the record.
4. The server is killed before the file is actually renamed or deleted.

We will prevent this race condition by ensuring that a log checkpoint
cannot occur between the durable write and the file system operation:

1. Durably write the FILE_DELETE or FILE_RENAME record.
2. Perform the file system operation.
3. Allow any log checkpoint to proceed.

mtr_t::commit_file(): Implement the DELETE or RENAME logic.

fil_delete_tablespace(): Delegate some of the logic to
mtr_t::commit_file().

fil_space_t::rename(): Delegate some logic to mtr_t::commit_file().
Remove the debug injection point fil_rename_tablespace_failure_2
because we do test RENAME failures without any debug injection.

fil_name_write_rename_low(), fil_name_write_rename(): Remove.

Tested by Matthias Leich
2022-06-21 16:59:21 +03:00
Marko Mäkelä
55f02c24a6 MDEV-28845 fixup: Prevent an infinite loop
buf_page_create_low(): Before retrying, release the exclusive page latch
in order to prevent an infinite loop in buf_pool_t::corrupted_evict().
2022-06-21 14:40:40 +03:00
Sergei Golubchik
2643aa43ae fix spider.variable_deprecation test 2022-06-16 22:39:35 +02:00
Marko Mäkelä
a8c22dae8b Merge 10.6 into 10.7 2022-06-16 10:50:58 +03:00
Marko Mäkelä
e99ba4ac8d MDEV-28864 Assertion `trx_id <= create_id' failed in innodb_check_version()
ha_innobase::prepare_inplace_alter_table(): If the ALTER TABLE
operation is no-op for InnoDB, do reset m_prebuilt->trx_id
so that ha_innobase::table_version() will always report either 0
or the identifier of the transaction that would commit changes
to the InnoDB data dictionary.

The failure scenario involved a completed DROP INDEX followed by
a no-op ALTER TABLE during which the server was killed.
An effort to create a reproducible test failed.
2022-06-16 09:53:46 +03:00
Shunsuke Tokunaga
c4f65d8fed
MDEV-21027 Assertion `part_share->auto_inc_initialized || !can_use_for_auto_inc_init()' failed in ha_partition::set_auto_increment_if_higher
ha_partition::set_auto_increment_if_higher expects
part_share->auto_inc_initialized is true or can_use_for_auto_inc_init()
is false (but as the comment of this method says, it returns false
only if we use Spider engine with DROP TABLE or ALTER TABLE query).
However, part_share->auto_inc_initialized becomes true only after all
partitions are opened (since 6dce6aeceb).

Therefore, I added a conditional expression in order to read all
partitions when we execute REPLACE on a table that has an
AUTO_INCREMENT column.           

Reviewed by: Nayuta Yanagisawa
Reviewed by: Alexey Botchkov
2022-06-16 13:28:24 +09:00
Marko Mäkelä
253806dffc MDEV-28845 InnoDB: Failing assertion: bpage->can_relocate() in buf0lru.cc
Since commit 0b47c126e3 (MDEV-13542)
we treat all-zero pages as corrupted ones.

During a stress test, a read-ahead of an all-zero page was triggered
and the page read was completed concurrently with buf_page_create_low().
This caused the assertion to fail, because buf_page_create_low() was
waiting for the page latch.

buf_page_get_low(): Only invoke buf_pool_t::corrupted_evict()
if the block was not already marked as corrupted.

buf_page_create_low(): On page identifier mismatch, retry the
buf_pool.page_hash lookup.

buf_pool_t::corrupted_evict(): Set the state of the block to FREED
so that a concurrent buf_page_get_low() will refuse to load the page.
Wait for the page latch to be vacant before proceeding to remove
the block from buf_pool.page_hash and buf_pool.LRU.

page_id_t::set_corrupted(), page_id_t::is_corrupted(): Accessors
for indicating a corrupted page identifier.

Tested by Matthias Leich
2022-06-15 17:00:05 +03:00