Commit graph

26308 commits

Author SHA1 Message Date
Marko Mäkelä
cb364a78d6 MDEV-31619 dict_stats_persistent_storage_check() may show garbage during --bootstrap
dict_stats_persistent_storage_check(): Do not output errmsg
if opt_bootstrap holds, because the message buffer would likely
be uninitialized.
2023-07-04 15:24:57 +03:00
Marko Mäkelä
f7b8a2c953 MDEV-31607 ER_DUP_KEY in mysql.innodb_table_stats upon RENAME on sequence
ha_innobase::delete_table(): Also on DROP SEQUENCE, do try to drop any
persistent statistics. They should really not be created for
SEQUENCE objects (which internally are 1-row no-rollback tables),
but that is how happened to always work.
2023-07-03 16:47:58 +03:00
Marko Mäkelä
dc1bd1802a MDEV-31386 InnoDB: Failing assertion: page_type == i_s_page_type[page_type].type_value
i_s_innodb_buffer_page_get_info(): Correct a condition.
After crash recovery, there may be some buffer pool pages in FREED state,
containing garbage (invalid data page contents). Let us ignore such pages
in the INFORMATION_SCHEMA output.

The test innodb.innodb_defragment_fill_factor will be removed, because
the queries that it is invoking on information_schema.innodb_buffer_page
would start to fail. The defragmentation feature was removed in
commit 7ca89af6f8 in MariaDB Server 11.1.

Tested by: Matthias Leich
2023-07-03 14:39:29 +03:00
Marko Mäkelä
3d90143859 MDEV-31559 btr_search_hash_table_validate() does not check if CHECK TABLE is killed
btr_search_hash_table_validate(), btr_search_validate(): Add the
parameter THD for checking if the statement has been killed.
Any non-QUICK CHECK TABLE will validate the entire adaptive hash index
for all InnoDB tables, which may be extremely slow when running
multiple concurrent CHECK TABLE.
2023-06-30 17:07:21 +03:00
Vlad Lesin
687fd6bef5 MDEV-30648 btr_estimate_n_rows_in_range() accesses unfixed, unlatched page
The issue is caused by MDEV-30400 fix.

There are two cursors in btr_estimate_n_rows_in_range() - p1 and p2, but
both share the same mtr. Each cursor contains mtr savepoint for the
previously fetched block to release it then the current block is
fetched.

Before MDEV-30400 the block was released with
mtr_t::release_block_at_savepoint(), it just unfixed a block and
released its page patch. In MDEV-30400 it was replaced with
mtr_t::rollback_to_savepoint(), which does the same as the former
mtr_t::release_block_at_savepoint(ulint begin, ulint end) but also
erases the corresponding slots from mtr memo, what invalidates any
stored mtr's memo savepoints, greater or equal to "begin".

The idea of the fix is to get rid of savepoints at all in
btr_estimate_n_rows_in_range() and
btr_estimate_n_rows_in_range_on_level(). As
mtr_t::rollback_to_savepoint() erases elements from mtr_t::m_memo, we
know what element of mtr_t::m_memo can be deleted on the certain case,
so there is no need to store savepoints.

See also the following slides for details:
https://docs.google.com/presentation/d/1RFYBo7EUhM22ab3GOYctv3j_3yC0vHtBY9auObZec8U

Reviewed by: Marko Mäkelä
2023-06-28 11:00:02 +03:00
Marko Mäkelä
694ce0d08e Merge 10.5 into 10.6 2023-06-27 13:03:32 +03:00
Marko Mäkelä
84dbd0253d MDEV-31487: Recovery or backup failure after innodb_undo_log_truncate=ON
recv_sys_t::parse(): For undo tablespace truncation mini-transactions,
remember the start_lsn instead of the end LSN. This is what we expect
after commit 461402a564 (MDEV-30479).
2023-06-27 09:12:38 +03:00
Marko Mäkelä
493083833b Merge 10.5 into 10.6 2023-06-26 17:11:38 +03:00
Vicentiu Ciorbaru
f20d556f19 Fix use of uninitialized variable
The original code generated a warning in gcc 13.1
2023-06-25 16:15:08 +03:00
Thirunarayanan Balathandayuthapani
bd076d4dff MDEV-31442 page_cleaner thread aborts while releasing the tablespace
- InnoDB shouldn't acquire the tablespace when it is being stopped
or closed
2023-06-16 14:58:48 +05:30
Thirunarayanan Balathandayuthapani
841e905f20 MDEV-31442 page_cleaner thread aborts while releasing the tablespace
After further I/O on a tablespace has been stopped
(for example due to DROP TABLE or an operation that
rebuilds a table), page cleaner thread tries to
flush the pending writes for the tablespace and
releases the tablespace reference even though it was not
acquired.

fil_space_t::flush(): Don't release the tablespace when it is
being stopped and closed

Thanks to Marko Mäkelä for suggesting this patch.
2023-06-09 18:15:33 +05:30
Thirunarayanan Balathandayuthapani
bf0a54df34 MDEV-31416 ASAN errors in dict_v_col_t::detach upon adding key to virtual column
- InnoDB throws ASAN error while adding the index on virtual column
of system versioned table. InnoDB wrongly assumes that virtual
column collation type changes, creates new column with different
character set. This leads to failure while detaching the column
from indexes.
2023-06-08 16:34:45 +05:30
Marko Mäkelä
80585c9d6f Merge 10.5 into 10.6 2023-06-08 10:42:56 +03:00
Marko Mäkelä
c25b496724 MDEV-31382 SET GLOBAL innodb_undo_log_truncate=ON has no effect on logically empty undo logs
innodb_undo_log_truncate_update(): A callback function. If
SET GLOBAL innodb_undo_log_truncate=ON, invoke
srv_wake_purge_thread_if_not_active().

srv_wake_purge_thread_if_not_active(): If innodb_undo_log_truncate=ON,
always wake up the purge subsystem.

srv_do_purge(): If the history is empty, invoke
trx_purge_truncate_history() in order to free undo log pages.

trx_purge_truncate_history(): If head.trx_no==0, consider the
cached undo logs to be free.

trx_purge(): Remove the parameter "bool truncate" and let the
caller invoke trx_purge_truncate_history() directly.

Reviewed by: Vladislav Lesin
2023-06-08 09:18:21 +03:00
Marko Mäkelä
3e40f9a7f3 MDEV-31355 innodb_undo_log_truncate=ON fails to wait for purge of enough transaction history
purge_sys_t::sees(): Wrapper for view.sees().

trx_purge_truncate_history(): Invoke purge_sys.sees() instead of
comparing to head.trx_no, to determine if undo pages can be safely freed.

The test innodb.cursor-restore-locking was adjusted by Vladislav Lesin,
as was the the debug instrumentation in row_purge_del_mark().

Reviewed by: Vladislav Lesin
2023-06-08 09:17:52 +03:00
Oleksandr Byelkin
04f0b955dd Merge branch '10.6' into 10.6.14 2023-06-07 19:59:52 +02:00
Marko Mäkelä
c93754d45e MDEV-31234 related cleanup
trx_purge_free_segment(), trx_purge_truncate_rseg_history():
Replace some unreachable code with debug assertions.
A buffer-fix does prevent pages from being evicted
from the buffer pool; see buf_page_t::can_relocate().

Tested by: Matthias Leich
2023-06-05 18:53:20 +02:00
Sergei Golubchik
a42a6fa99b Merge branch 'bb-10.5-release' into bb-10.6-release 2023-06-05 18:53:02 +02:00
Marko Mäkelä
cf37e44eec MDEV-31350: Hang in innodb.recovery_memory
buf_flush_page_cleaner(): Whenever buf_pool.ran_out(), invoke
buf_pool.get_oldest_modification(0) so that all clean blocks
will be removed from buf_pool.flush_list and buf_flush_LRU_list_batch()
will be able to evict some pages.

This fixes a regression that was likely caused by
commit a55b951e60 (MDEV-26827).
2023-06-03 11:12:32 +02:00
Marko Mäkelä
dd298873da MDEV-31309 Innodb_buffer_pool_read_requests is not updated correctly
srv_export_innodb_status(): Update
export_vars.innodb_buffer_pool_read_requests as it was done
before commit a55b951e60 (MDEV-26827).
If innodb_status_variables[] pointed to a sharded variable, it would
only access the first shard.
2023-06-03 11:12:27 +02:00
Marko Mäkelä
89eb6fa8a7 MDEV-31308 InnoDB monitor trx_rseg_history_len was accidentally disabled by default
innodb_counter_info[]: Revert a change that was accidentally made in
commit 204e7225dc
2023-06-03 11:12:21 +02:00
Marko Mäkelä
883333a74e MDEV-31158: Potential hang with ROW_FORMAT=COMPRESSED tables
btr_cur_need_opposite_intention(): Check also page_zip_available()
so that we will escalate to exclusive index latch when a non-leaf
page may have to be split further due to ROW_FORMAT=COMPRESSED page
overflow.

Tested by: Matthias Leich
2023-06-03 11:12:16 +02:00
Marko Mäkelä
459eb9a686 MDEV-29593 fixup: Avoid a leak if rseg.undo_cached is corrupted
trx_purge_truncate_rseg_history(): Avoid a leak similar to the one
that was fixed in MDEV-31324, in case a supposedly cached undo log
page is not found in the rseg.undo_cached list.
2023-06-03 11:12:11 +02:00
Marko Mäkelä
e89bd39c9b MDEV-31343 Another server hang with innodb_undo_log_truncate=ON
trx_purge_truncate_history(): While waiting for a write-fixed block
to become available, simply wait for an exclusive latch on it.
Also, simplify the iteration: first check for oldest_modification>2
(to ignore clean pages or pages belonging to the temporary tablespace)
and then compare the tablespace identifier.

Before releasing buf_pool.flush_list_mutex we will buffer-fix the block
of interest. In that way, buf_page_t::can_relocate() will not hold on
the block and it must remain in the buffer pool until we have acquired
an exclusive latch on it. If the block is still dirty, we will register
it with the tablespace truncation mini-transaction; else, we will simply
release the latch and buffer-fix and move to the next block.

This also reverts commit c4d7939989
because that fix should no longer be necessary; the wait for an
exclusive block latch should allow buf_pool_t::release_freed_page()
on the same block to proceed.

Tested by: Axel Schwenke, Matthias Leich
2023-06-03 11:12:03 +02:00
Marko Mäkelä
3b4b512d8e MDEV-31234 fixup: Allow innodb_undo_log_truncate=ON after upgrade
trx_purge_truncate_history(): Relax a condition that would prevent
undo log truncation if the undo log tablespaces were "contaminated"
by the bug that commit e0084b9d31 fixed.
That is, trx_purge_truncate_rseg_history() would have invoked
flst_remove() on TRX_RSEG_HISTORY but not reduced TRX_RSEG_HISTORY_SIZE.

To avoid any regression with normal operation, we implement this
fixup during slow shutdown only. The condition on the history list
being empty is necessary: without it, in the test
innodb.undo_truncate_recover there may be much fewer than the
expected 90,000 calls to row_purge() before the truncation.
That is, we would truncate the undo tablespace before actually having
processed all undo log records in it.

To truncate such "contaminated" or "bloated" undo log tablespaces
(when using innodb_undo_tablespaces=2 or more)
you can execute the following SQL:

BEGIN;INSERT mysql.innodb_table_stats VALUES('','',DEFAULT,0,0,0);ROLLBACK;
SET GLOBAL innodb_undo_log_truncate=ON, innodb_fast_shutdown=0;
SHUTDOWN;

The first line creates a dummy InnoDB transaction, to ensure that there
will be some history to be purged during shutdown and that the undo
tablespaces will be truncated.
2023-06-03 10:59:53 +02:00
Marko Mäkelä
48d6a5f61b MDEV-31234 fixup: Free some UNDO pages earlier
trx_purge_truncate_rseg_history(): Add a parameter to specify if
the entire rollback segment is safe to be freed. If not, we may
still be able to invoke trx_undo_truncate_start() and free some pages.
2023-06-03 10:59:47 +02:00
Marko Mäkelä
318012a80a MDEV-31234 InnoDB does not free UNDO after the fix of MDEV-30671
trx_purge_truncate_history(): Only call trx_purge_truncate_rseg_history()
if the rollback segment is safe to process. This will avoid leaking undo
log pages that are not yet ready to be processed. This fixes a regression
that was introduced in
commit 0de3be8cfd (MDEV-30671).

trx_sys_t::any_active_transactions(): Separately count XA PREPARE
transactions.

srv_purge_should_exit(): Terminate slow shutdown if the history size
does not change and XA PREPARE transactions exist in the system.
This will avoid a hang of the test innodb.recovery_shutdown.

Tested by: Matthias Leich
2023-06-03 10:59:42 +02:00
Marko Mäkelä
f569e06e03 MDEV-31385 Change buffer stale entries leads to corruption while reusing page
buf_page_free(): If buffered changes existed for the page, drop them.

Co-developed with Thirunarayanan Balathandayuthapani
2023-06-02 11:06:09 +03:00
Marko Mäkelä
8a86df37ef MDEV-31088 Server freeze due to innodb_change_buffering
A 3-thread deadlock has been frequently observed when using
innodb_change_buffering!=none and innodb_file_per_table=0:

(1) ibuf_merge_or_delete_for_page() holding an exclusive latch on the block
and waiting for an exclusive tablespace latch in fseg_page_is_allocated()
(2) btr_free_but_not_root() in fseg_free_step() waiting for an
exclusive tablespace latch
(3) fsp_alloc_free_page() holding the exclusive tablespace latch and waiting
for a latch on the block, which it is reallocating for something else

While this was reproduced using innodb_file_per_table=0, this hang should
be theoretically possible in .ibd files as well, when the recovery or
cleanup of a failed DROP INDEX or ADD INDEX is executing concurrently
with something that involves page allocation.

ibuf_merge_or_delete_for_page(): Avoid invoking fseg_page_is_allocated()
when block==nullptr. The call was redundant in this case, and it could
cause deadlocks due to latching order violation.

ibuf_read_merge_pages(): Acquire an exclusive tablespace latch
before invoking buf_page_get_gen(), which may cause
fseg_page_is_allocated() to be invoked in ibuf_merge_or_delete_for_page().

Note: This will not fix all latching order violations in this area!
Deadlocks involving ibuf_merge_or_delete_for_page(block!=nullptr) are
still possible if the caller is not acquiring an exclusive tablespace latch
upfront. This would be the case in any read operation that involves a
change buffer merge, such as SELECT, CHECK TABLE, or any DML operation that
cannot be buffered in the change buffer.
2023-06-02 10:44:34 +03:00
Marko Mäkelä
548a41c5ec Merge 10.5 into 10.6 2023-06-01 12:28:40 +03:00
Marko Mäkelä
bb9da13baf MDEV-31373 innodb_undo_log_truncate=ON recovery results in a corrupted undo log
recv_sys_t::apply(): When applying an undo log truncation operation,
invoke os_file_truncate() on space->recv_size, which must not be
less than the original truncated file size.

Alternatively, as pointed out by Thirunarayanan Balathandayuthapani,
we could assign space->size = t.pages, so that
fil_system_t::extend_to_recv_size() would extend the file back
to space->recv_size.
2023-06-01 12:11:18 +03:00
Marko Mäkelä
3aea77edeb MDEV-31347 fil_ibd_create() may hijack the file handle of an old file
fil_space_t::add(): If a file handle was passed, invoke
fil_node_t::find_metadata() before releasing fil_system.mutex.
The call was moved from fil_ibd_create().

This is a 10.5 version of commit e3b06156c6
from 10.6.
2023-06-01 09:41:17 +03:00
Thirunarayanan Balathandayuthapani
5919f7b675 MDEV-31264 Purge trying to access freed secondary index page
- InnoDB purge tries to access aborted secondary index and access
the freed secondary index root page.
2023-05-31 19:07:41 +05:30
Marko Mäkelä
e3b06156c6 MDEV-31347 fil_ibd_create() may hijack the file handle of an old file
fil_ibd_create(): Hold fil_system.mutex until fil_node_t::find_metadata()
has completed, so that node->handle cannot be closed by a concurrent
thread. This race condition was introduced
in commit 10dd290b4b (MDEV-17380).

Tested by: Matthias Leich
2023-05-31 15:25:07 +03:00
Marko Mäkelä
eb20e7c900 MDEV-31353 InnoDB recovery hangs after reporting corruption
recv_recover_page(): Remove some code which was added in
commit 0b47c126e3 with
no good reason and which would cause a hang after a corrupted
page was reported during crash recovery.

Tested by: Matthias Leich
2023-05-31 15:20:54 +03:00
Marko Mäkelä
a6c0a27696 MDEV-31362 recv_sys_t::apply(bool): Assertion `!last_batch || recovered_lsn == scanned_lsn' failed
recv_sys_t::apply(): Remove a bogus debug assertion that had been
added in commit f2c17cc9d9 (MDEV-29911).

It is perfectly normal that when the server was killed in the middle of
writing multiple redo log blocks, the recovery would end such that
recv_sys.scanned_lsn will point to the end of the last complete 512-byte
log block, but recv_sys.recovered_lsn will be less than that.

Also, correct the function comment of recv_sys_t::parse().
2023-05-30 17:21:49 +03:00
Marko Mäkelä
ce547cfc05 MDEV-31350: Hang in innodb.recovery_memory
buf_flush_page_cleaner(): Whenever buf_pool.ran_out(), invoke
buf_pool.get_oldest_modification(0) so that all clean blocks
will be removed from buf_pool.flush_list and buf_flush_LRU_list_batch()
will be able to evict some pages.

This fixes a regression that was likely caused by
commit a55b951e60 (MDEV-26827).
2023-05-26 16:40:07 +03:00
Marko Mäkelä
7b72fc0a57 MDEV-22739 !cursor->index->is_committed() in row0ins.cc
row_ins_sec_index_entry_by_modify(): When noticing a corrupted secondary
index on which CREATE INDEX is not in progress, return DB_CORRUPTION
instead of intentionally crashing the server.

Tested by: Matthias Leich
2023-05-26 16:40:02 +03:00
Marko Mäkelä
e38c075aa0 MDEV-31346 trx_purge_add_undo_to_history() is not optimal
trx_undo_set_state_at_finish(): Merge to its only caller,
trx_purge_add_undo_to_history().

trx_purge_add_undo_to_history(): Evaluate the condition related to
TRX_UNDO_STATE only once.

Tested by: Matthias Leich
2023-05-26 16:39:46 +03:00
Marko Mäkelä
db8765500e MDEV-31343 Another server hang with innodb_undo_log_truncate=ON
trx_purge_truncate_history(): While waiting for a write-fixed block
to become available, simply wait for an exclusive latch on it.
Also, simplify the iteration: first check for oldest_modification>2
(to ignore clean pages or pages belonging to the temporary tablespace)
and then compare the tablespace identifier.

Before releasing buf_pool.flush_list_mutex we will buffer-fix the block
of interest. In that way, buf_page_t::can_relocate() will not hold on
the block and it must remain in the buffer pool until we have acquired
an exclusive latch on it. If the block is still dirty, we will register
it with the tablespace truncation mini-transaction; else, we will simply
release the latch and buffer-fix and move to the next block.

This also reverts commit c4d7939989
because that fix should no longer be necessary; the wait for an
exclusive block latch should allow buf_pool_t::release_freed_page()
on the same block to proceed.

Tested by: Axel Schwenke, Matthias Leich
2023-05-26 16:16:10 +03:00
Alexander Barkov
9edb1a5ce3 MDEV-30483 After upgrade to 10.6 from Mysql 5.7 seeing "InnoDB: Column last_update in table mysql.innodb_table_stats is BINARY(4) NOT NULL but should be INT UNSIGNED NOT NULL"
Problem:

Field_timestampf implementations differ in MySQL and MariaDB:
- MariaDB sets the UNSIGNED_FLAG in Field::flags
- MySQL does not

The reference table structures
(defined in table_stats_schema and index_stats_schema)
expected the last_update column to have the DATA_UNSIGNED flag,
because MariaDB's Field_timestampf has the UNSIGNED_FLAG.
It worked fine on pure MariaDB installations.

However, if a MariaDB server starts over a MySQL-5.7 data directory during
a migration, the last_update column does not have DATA_UNSIGNED flag,
because MySQL's Field_timestampf does not have the UNSIGNED_FLAG.

This made InnoDB (after the migration from MySQL) complain into the server
error log about the unexpected data type.

The actual fix is done in storage/innobase/dict/dict0stats.cc:
It removes DATA_UNSIGNED from the prtype_mask member of the reference columns,
so now it does not require the underlying columns to have this flag.

The rest of the fix is needed for MTR tests.
The new data type plugin TYPE_MYSQL_TIMESTAMP implements a slightly modified
version of Field_timestampf, which removes the unsigned flag, so
it works like MySQL's Field_timestampf.

The MTR test ALTERs the data type of the columns
table_stats_schema.last_update and index_stats_schema.last_update
from TIMESTAMP to TYPE_MYSQL_TIMESTAMP, then makes InnoDB
verify the structure of the two statistics tables by creating
and populating an InnoDB table t1.

Without the fix made storage/innobase/dict/dict0stats.cc,
MTR complains about unexpected warnings in the server error log:

[ERROR] InnoDB: Column last_update in table mysql.innodb_table_stats is ...
[ERROR] InnoDB: Column last_update in table mysql.innodb_index_stats is ...

With the fix made storage/innobase/dict/dict0stats.cc these warnings
go away.
2023-05-25 05:25:39 +04:00
Thirunarayanan Balathandayuthapani
7737f15f87 MDEV-31333 fsp_free_page() fails to move the extent from
FSP_FREE_FRAG to FSP_FREE list

- This issue was caused by commit 0b47c126e3.
In fsp_free_page(), InnoDB should set XDES_FREE_BIT of the page before
moving the extent from FSP_FREE_FRAG to FSP_FREE list.
2023-05-24 15:15:29 +05:30
Marko Mäkelä
b220bb756b Merge bb-10.6-release into 10.6 2023-05-24 08:37:19 +03:00
Marko Mäkelä
98de15aba1 Merge bb-10.5-release into bb-10.6-release 2023-05-24 08:36:30 +03:00
Marko Mäkelä
383105dae1 Merge bb-10.5-release into 10.5 2023-05-24 08:28:20 +03:00
Marko Mäkelä
c5cf94b2dc MDEV-31234 fixup: Free some UNDO pages earlier
trx_purge_truncate_rseg_history(): Add a parameter to specify if
the entire rollback segment is safe to be freed. If not, we may
still be able to invoke trx_undo_truncate_start() and free some pages.
2023-05-24 08:25:26 +03:00
Marko Mäkelä
270eeeb523 Merge 10.5 into 10.6 2023-05-23 12:25:39 +03:00
Marko Mäkelä
9c35f9c9c1 MDEV-31234 fixup: Allow innodb_undo_log_truncate=ON after upgrade
trx_purge_truncate_history(): Relax a condition that would prevent
undo log truncation if the undo log tablespaces were "contaminated"
by the bug that commit e0084b9d31 fixed.
That is, trx_purge_truncate_rseg_history() would have invoked
flst_remove() on TRX_RSEG_HISTORY but not reduced TRX_RSEG_HISTORY_SIZE.

To avoid any regression with normal operation, we implement this
fixup during slow shutdown only. The condition on the history list
being empty is necessary: without it, in the test
innodb.undo_truncate_recover there may be much fewer than the
expected 90,000 calls to row_purge() before the truncation.
That is, we would truncate the undo tablespace before actually having
processed all undo log records in it.

To truncate such "contaminated" or "bloated" undo log tablespaces
(when using innodb_undo_tablespaces=2 or more)
you can execute the following SQL:

BEGIN;INSERT mysql.innodb_table_stats VALUES('','',DEFAULT,0,0,0);ROLLBACK;
SET GLOBAL innodb_undo_log_truncate=ON, innodb_fast_shutdown=0;
SHUTDOWN;

The first line creates a dummy InnoDB transaction, to ensure that there
will be some history to be purged during shutdown and that the undo
tablespaces will be truncated.
2023-05-23 12:20:27 +03:00
Monty
cd37e49422 MDEV-31083 ASAN use-after-poison in myrg_attach_children
The reason for ASAN report was that the MERGE and MYISAM file
had different key definitions, which is not allowed.

Fixed by ensuring that the MERGE code is not copying more key stats
than what is in the MyISAM file.

Other things:
- Give an error if different MyISAM files has different number of
  key parts.
2023-05-23 09:16:36 +03:00
Monty
6a0314063d Make install.db read only in mtr
This ensures that no mtr test can change install.db after it's initial
creation as changing it while as another thread is coping it will lead to
failures in at least InnoDB and Aria recovery.

Fixed spider/bugfix.mdev_30370 that was wrongly used install.db
2023-05-23 09:16:36 +03:00