Commit graph

197939 commits

Author SHA1 Message Date
Sergei Petrunia
928012a27a MDEV-31403: Server crashes in st_join_table::choose_best_splitting
The code in choose_best_splitting() assumed that the join prefix is
in join->positions[].

This is not necessarily the case. This function might be called when
the join prefix is in join->best_positions[], too.
Follow the approach from best_access_path(), which calls this function:
pass the current join prefix as an argument,
"const POSITION *join_positions" and use that.
2023-06-05 18:24:39 +03:00
Otto Kekalainen
fcfd6361cb Deb: Fix blocksize check to use $mysql_datadir/$datadir correctly
In commit f99a8918 this line was changed to not use awk, and new version
copied both to init file and preinst file but overlooking that they use
different variable names.

Also fix minor syntax issues to make Shellcheck happy.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.
2023-06-03 11:12:37 +02:00
Marko Mäkelä
cf37e44eec MDEV-31350: Hang in innodb.recovery_memory
buf_flush_page_cleaner(): Whenever buf_pool.ran_out(), invoke
buf_pool.get_oldest_modification(0) so that all clean blocks
will be removed from buf_pool.flush_list and buf_flush_LRU_list_batch()
will be able to evict some pages.

This fixes a regression that was likely caused by
commit a55b951e60 (MDEV-26827).
2023-06-03 11:12:32 +02:00
Marko Mäkelä
dd298873da MDEV-31309 Innodb_buffer_pool_read_requests is not updated correctly
srv_export_innodb_status(): Update
export_vars.innodb_buffer_pool_read_requests as it was done
before commit a55b951e60 (MDEV-26827).
If innodb_status_variables[] pointed to a sharded variable, it would
only access the first shard.
2023-06-03 11:12:27 +02:00
Marko Mäkelä
89eb6fa8a7 MDEV-31308 InnoDB monitor trx_rseg_history_len was accidentally disabled by default
innodb_counter_info[]: Revert a change that was accidentally made in
commit 204e7225dc
2023-06-03 11:12:21 +02:00
Marko Mäkelä
883333a74e MDEV-31158: Potential hang with ROW_FORMAT=COMPRESSED tables
btr_cur_need_opposite_intention(): Check also page_zip_available()
so that we will escalate to exclusive index latch when a non-leaf
page may have to be split further due to ROW_FORMAT=COMPRESSED page
overflow.

Tested by: Matthias Leich
2023-06-03 11:12:16 +02:00
Marko Mäkelä
459eb9a686 MDEV-29593 fixup: Avoid a leak if rseg.undo_cached is corrupted
trx_purge_truncate_rseg_history(): Avoid a leak similar to the one
that was fixed in MDEV-31324, in case a supposedly cached undo log
page is not found in the rseg.undo_cached list.
2023-06-03 11:12:11 +02:00
Marko Mäkelä
e89bd39c9b MDEV-31343 Another server hang with innodb_undo_log_truncate=ON
trx_purge_truncate_history(): While waiting for a write-fixed block
to become available, simply wait for an exclusive latch on it.
Also, simplify the iteration: first check for oldest_modification>2
(to ignore clean pages or pages belonging to the temporary tablespace)
and then compare the tablespace identifier.

Before releasing buf_pool.flush_list_mutex we will buffer-fix the block
of interest. In that way, buf_page_t::can_relocate() will not hold on
the block and it must remain in the buffer pool until we have acquired
an exclusive latch on it. If the block is still dirty, we will register
it with the tablespace truncation mini-transaction; else, we will simply
release the latch and buffer-fix and move to the next block.

This also reverts commit c4d7939989
because that fix should no longer be necessary; the wait for an
exclusive block latch should allow buf_pool_t::release_freed_page()
on the same block to proceed.

Tested by: Axel Schwenke, Matthias Leich
2023-06-03 11:12:03 +02:00
Marko Mäkelä
3b4b512d8e MDEV-31234 fixup: Allow innodb_undo_log_truncate=ON after upgrade
trx_purge_truncate_history(): Relax a condition that would prevent
undo log truncation if the undo log tablespaces were "contaminated"
by the bug that commit e0084b9d31 fixed.
That is, trx_purge_truncate_rseg_history() would have invoked
flst_remove() on TRX_RSEG_HISTORY but not reduced TRX_RSEG_HISTORY_SIZE.

To avoid any regression with normal operation, we implement this
fixup during slow shutdown only. The condition on the history list
being empty is necessary: without it, in the test
innodb.undo_truncate_recover there may be much fewer than the
expected 90,000 calls to row_purge() before the truncation.
That is, we would truncate the undo tablespace before actually having
processed all undo log records in it.

To truncate such "contaminated" or "bloated" undo log tablespaces
(when using innodb_undo_tablespaces=2 or more)
you can execute the following SQL:

BEGIN;INSERT mysql.innodb_table_stats VALUES('','',DEFAULT,0,0,0);ROLLBACK;
SET GLOBAL innodb_undo_log_truncate=ON, innodb_fast_shutdown=0;
SHUTDOWN;

The first line creates a dummy InnoDB transaction, to ensure that there
will be some history to be purged during shutdown and that the undo
tablespaces will be truncated.
2023-06-03 10:59:53 +02:00
Marko Mäkelä
48d6a5f61b MDEV-31234 fixup: Free some UNDO pages earlier
trx_purge_truncate_rseg_history(): Add a parameter to specify if
the entire rollback segment is safe to be freed. If not, we may
still be able to invoke trx_undo_truncate_start() and free some pages.
2023-06-03 10:59:47 +02:00
Marko Mäkelä
318012a80a MDEV-31234 InnoDB does not free UNDO after the fix of MDEV-30671
trx_purge_truncate_history(): Only call trx_purge_truncate_rseg_history()
if the rollback segment is safe to process. This will avoid leaking undo
log pages that are not yet ready to be processed. This fixes a regression
that was introduced in
commit 0de3be8cfd (MDEV-30671).

trx_sys_t::any_active_transactions(): Separately count XA PREPARE
transactions.

srv_purge_should_exit(): Terminate slow shutdown if the history size
does not change and XA PREPARE transactions exist in the system.
This will avoid a hang of the test innodb.recovery_shutdown.

Tested by: Matthias Leich
2023-06-03 10:59:42 +02:00
Sergei Golubchik
eb472f77e3 Revert "MDEV-30473 : Do not allow GET_LOCK() / RELEASE_LOCK() in cluster"
This reverts commit b05218e08f.
2023-06-03 10:39:34 +02:00
Sergei Golubchik
0fd54c9892 Revert "MDEV-30473 : Do not allow GET_LOCK() / RELEASE_LOCK() in cluster"
This reverts commit 844ddb1109.

This fixes MDEV-30967, MDEV-31325, MDEV-31388
2023-06-03 10:39:34 +02:00
Igor Babaev
8f3bf593d2 MDEV-31240 Crash with condition pushable into derived and containing outer reference
This bug could affect queries containing a subquery over splittable derived
tables and having an outer references in its WHERE clause. If such subquery
contained an equality condition whose left part was a reference to a column
of the derived table and the right part referred only to outer columns
then the server crashed in the function st_join_table::choose_best_splitting()
The crashing code was added in the commit ce7ffe61d8
that made the code of the function sensitive to presence of the flag
OUTER_REF_TABLE_BIT in the KEYUSE_EXT::needed_in_prefix fields.

The field needed_in_prefix of the KEYUSE_EXT structure should not contain
table maps with OUTER_REF_TABLE_BIT or RAND_TABLE_BIT.

Note that this fix is quite conservative: for affected queries it just
returns the query plans that were used before the above mentioned commit.
In fact the equalities causing crashes should be pushed into derived tables
without any usage of split optimization.

Approved by Sergei Petrunia <sergey@mariadb.com>
2023-06-03 10:39:34 +02:00
Igor Babaev
aa713f5ae2 MDEV-31224 Crash with EXPLAIN EXTENDED for multi-table update of system table
EXPLAIN EXTENDED should always print the field item used in the left part
of an equality expression from the SET clause of an update statement as a
reference to table column.

Approved by Oleksandr Byelkin <sanja@mariadb.com>
2023-06-03 10:39:34 +02:00
Daniel Bartholomew
54324e542f bump the VERSION 2023-06-03 10:39:34 +02:00
Marko Mäkelä
f569e06e03 MDEV-31385 Change buffer stale entries leads to corruption while reusing page
buf_page_free(): If buffered changes existed for the page, drop them.

Co-developed with Thirunarayanan Balathandayuthapani
2023-06-02 11:06:09 +03:00
Marko Mäkelä
8a86df37ef MDEV-31088 Server freeze due to innodb_change_buffering
A 3-thread deadlock has been frequently observed when using
innodb_change_buffering!=none and innodb_file_per_table=0:

(1) ibuf_merge_or_delete_for_page() holding an exclusive latch on the block
and waiting for an exclusive tablespace latch in fseg_page_is_allocated()
(2) btr_free_but_not_root() in fseg_free_step() waiting for an
exclusive tablespace latch
(3) fsp_alloc_free_page() holding the exclusive tablespace latch and waiting
for a latch on the block, which it is reallocating for something else

While this was reproduced using innodb_file_per_table=0, this hang should
be theoretically possible in .ibd files as well, when the recovery or
cleanup of a failed DROP INDEX or ADD INDEX is executing concurrently
with something that involves page allocation.

ibuf_merge_or_delete_for_page(): Avoid invoking fseg_page_is_allocated()
when block==nullptr. The call was redundant in this case, and it could
cause deadlocks due to latching order violation.

ibuf_read_merge_pages(): Acquire an exclusive tablespace latch
before invoking buf_page_get_gen(), which may cause
fseg_page_is_allocated() to be invoked in ibuf_merge_or_delete_for_page().

Note: This will not fix all latching order violations in this area!
Deadlocks involving ibuf_merge_or_delete_for_page(block!=nullptr) are
still possible if the caller is not acquiring an exclusive tablespace latch
upfront. This would be the case in any read operation that involves a
change buffer merge, such as SELECT, CHECK TABLE, or any DML operation that
cannot be buffered in the change buffer.
2023-06-02 10:44:34 +03:00
Marko Mäkelä
548a41c5ec Merge 10.5 into 10.6 2023-06-01 12:28:40 +03:00
Marko Mäkelä
bb9da13baf MDEV-31373 innodb_undo_log_truncate=ON recovery results in a corrupted undo log
recv_sys_t::apply(): When applying an undo log truncation operation,
invoke os_file_truncate() on space->recv_size, which must not be
less than the original truncated file size.

Alternatively, as pointed out by Thirunarayanan Balathandayuthapani,
we could assign space->size = t.pages, so that
fil_system_t::extend_to_recv_size() would extend the file back
to space->recv_size.
2023-06-01 12:11:18 +03:00
Marko Mäkelä
3aea77edeb MDEV-31347 fil_ibd_create() may hijack the file handle of an old file
fil_space_t::add(): If a file handle was passed, invoke
fil_node_t::find_metadata() before releasing fil_system.mutex.
The call was moved from fil_ibd_create().

This is a 10.5 version of commit e3b06156c6
from 10.6.
2023-06-01 09:41:17 +03:00
Thirunarayanan Balathandayuthapani
5919f7b675 MDEV-31264 Purge trying to access freed secondary index page
- InnoDB purge tries to access aborted secondary index and access
the freed secondary index root page.
2023-05-31 19:07:41 +05:30
Marko Mäkelä
e3b06156c6 MDEV-31347 fil_ibd_create() may hijack the file handle of an old file
fil_ibd_create(): Hold fil_system.mutex until fil_node_t::find_metadata()
has completed, so that node->handle cannot be closed by a concurrent
thread. This race condition was introduced
in commit 10dd290b4b (MDEV-17380).

Tested by: Matthias Leich
2023-05-31 15:25:07 +03:00
Marko Mäkelä
eb20e7c900 MDEV-31353 InnoDB recovery hangs after reporting corruption
recv_recover_page(): Remove some code which was added in
commit 0b47c126e3 with
no good reason and which would cause a hang after a corrupted
page was reported during crash recovery.

Tested by: Matthias Leich
2023-05-31 15:20:54 +03:00
Tuukka Pasanen
30fb72ca6e MDEV-31331: Fix cut'n'paste variable name in Debian pre-inst script
There is unwanted cut'n'paste variable name in Debian pre-inst
script which causes:

df: '': No such file or directory
/var/lib/dpkg/tmp.ci/preinst: line 215: [: : integer expression expected

Rename variable to correct one and make check that that directory
or symlink really exists. If it does not then fail with error
and message.
2023-05-31 10:26:19 +10:00
Marko Mäkelä
a6c0a27696 MDEV-31362 recv_sys_t::apply(bool): Assertion `!last_batch || recovered_lsn == scanned_lsn' failed
recv_sys_t::apply(): Remove a bogus debug assertion that had been
added in commit f2c17cc9d9 (MDEV-29911).

It is perfectly normal that when the server was killed in the middle of
writing multiple redo log blocks, the recovery would end such that
recv_sys.scanned_lsn will point to the end of the last complete 512-byte
log block, but recv_sys.recovered_lsn will be less than that.

Also, correct the function comment of recv_sys_t::parse().
2023-05-30 17:21:49 +03:00
Otto Kekalainen
ea66df2f45 Deb: Fix blocksize check to use $mysql_datadir/$datadir correctly
In commit f99a8918 this line was changed to not use awk, and new version
copied both to init file and preinst file but overlooking that they use
different variable names.

Also fix minor syntax issues to make Shellcheck happy.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.
2023-05-30 18:55:13 +10:00
Marko Mäkelä
ce547cfc05 MDEV-31350: Hang in innodb.recovery_memory
buf_flush_page_cleaner(): Whenever buf_pool.ran_out(), invoke
buf_pool.get_oldest_modification(0) so that all clean blocks
will be removed from buf_pool.flush_list and buf_flush_LRU_list_batch()
will be able to evict some pages.

This fixes a regression that was likely caused by
commit a55b951e60 (MDEV-26827).
2023-05-26 16:40:07 +03:00
Marko Mäkelä
7b72fc0a57 MDEV-22739 !cursor->index->is_committed() in row0ins.cc
row_ins_sec_index_entry_by_modify(): When noticing a corrupted secondary
index on which CREATE INDEX is not in progress, return DB_CORRUPTION
instead of intentionally crashing the server.

Tested by: Matthias Leich
2023-05-26 16:40:02 +03:00
Marko Mäkelä
e38c075aa0 MDEV-31346 trx_purge_add_undo_to_history() is not optimal
trx_undo_set_state_at_finish(): Merge to its only caller,
trx_purge_add_undo_to_history().

trx_purge_add_undo_to_history(): Evaluate the condition related to
TRX_UNDO_STATE only once.

Tested by: Matthias Leich
2023-05-26 16:39:46 +03:00
Marko Mäkelä
db8765500e MDEV-31343 Another server hang with innodb_undo_log_truncate=ON
trx_purge_truncate_history(): While waiting for a write-fixed block
to become available, simply wait for an exclusive latch on it.
Also, simplify the iteration: first check for oldest_modification>2
(to ignore clean pages or pages belonging to the temporary tablespace)
and then compare the tablespace identifier.

Before releasing buf_pool.flush_list_mutex we will buffer-fix the block
of interest. In that way, buf_page_t::can_relocate() will not hold on
the block and it must remain in the buffer pool until we have acquired
an exclusive latch on it. If the block is still dirty, we will register
it with the tablespace truncation mini-transaction; else, we will simply
release the latch and buffer-fix and move to the next block.

This also reverts commit c4d7939989
because that fix should no longer be necessary; the wait for an
exclusive block latch should allow buf_pool_t::release_freed_page()
on the same block to proceed.

Tested by: Axel Schwenke, Matthias Leich
2023-05-26 16:16:10 +03:00
Alexander Barkov
03a9366c73 Extra tests for MDEV-30483 After upgrade to 10.6 from Mysql 5.7 seeing "InnoDB: Column last_update in table mysql.innodb_table_stats is BINARY(4) NOT NULL but should be INT UNSIGNED NOT NULL"
Adding tests demonstrating that columns:
- mysql.innodb_table_stats.last_update
- mysql.innodb_index_stats.last_update

contain sane values close to NOW() rathar than a garbage.

Tests cover these three underlying TIMESTAMP data formats:

- MariaDB Field_timestamp0 - UINT4 based
  Like in a MariaDB native installation running with mysql56_temporal_format=0

- MariaDB Field_timestampf - BINARY(4) based, with UNSIGNED_FLAG
  Like in a MariaDB native installation running with mysql56_temporal_format=1

- MySQL-alike Field_timestampf - BINARY(4) based, without UNSIGNED_FLAG
  Like with a MariaDB server running over a MySQL-5.6 directory
  (e.g. during a migragion).
2023-05-26 16:47:16 +04:00
Oleksandr Byelkin
44c9008ba6 Merge branch '10.9' into bb-10.9-release 2023-05-25 11:35:05 +02:00
Marko Mäkelä
31be25349f Merge 10.6 into 10.9 2023-05-25 09:24:32 +03:00
Alexander Barkov
9edb1a5ce3 MDEV-30483 After upgrade to 10.6 from Mysql 5.7 seeing "InnoDB: Column last_update in table mysql.innodb_table_stats is BINARY(4) NOT NULL but should be INT UNSIGNED NOT NULL"
Problem:

Field_timestampf implementations differ in MySQL and MariaDB:
- MariaDB sets the UNSIGNED_FLAG in Field::flags
- MySQL does not

The reference table structures
(defined in table_stats_schema and index_stats_schema)
expected the last_update column to have the DATA_UNSIGNED flag,
because MariaDB's Field_timestampf has the UNSIGNED_FLAG.
It worked fine on pure MariaDB installations.

However, if a MariaDB server starts over a MySQL-5.7 data directory during
a migration, the last_update column does not have DATA_UNSIGNED flag,
because MySQL's Field_timestampf does not have the UNSIGNED_FLAG.

This made InnoDB (after the migration from MySQL) complain into the server
error log about the unexpected data type.

The actual fix is done in storage/innobase/dict/dict0stats.cc:
It removes DATA_UNSIGNED from the prtype_mask member of the reference columns,
so now it does not require the underlying columns to have this flag.

The rest of the fix is needed for MTR tests.
The new data type plugin TYPE_MYSQL_TIMESTAMP implements a slightly modified
version of Field_timestampf, which removes the unsigned flag, so
it works like MySQL's Field_timestampf.

The MTR test ALTERs the data type of the columns
table_stats_schema.last_update and index_stats_schema.last_update
from TIMESTAMP to TYPE_MYSQL_TIMESTAMP, then makes InnoDB
verify the structure of the two statistics tables by creating
and populating an InnoDB table t1.

Without the fix made storage/innobase/dict/dict0stats.cc,
MTR complains about unexpected warnings in the server error log:

[ERROR] InnoDB: Column last_update in table mysql.innodb_table_stats is ...
[ERROR] InnoDB: Column last_update in table mysql.innodb_index_stats is ...

With the fix made storage/innobase/dict/dict0stats.cc these warnings
go away.
2023-05-25 05:25:39 +04:00
Monty
9b3084b7be Fixed typo in xtrabackup.c 2023-05-24 18:39:55 +03:00
Monty
d77d9e1f6f MENT-1703 Repeatable crash during backup after processing very large ibdata1
The crash happened in filename_to_spacename() when using it on a
filename that is not in the format of "./database/table.ibd".
According to Marko, it is possible the function is called with
the path to an undo file, which would cause a crash.

This patch fixes this by,  instead of crashing with unexpected filenames,
returning them 'as such', except for changing all '\' to '/'.
2023-05-24 15:00:26 +03:00
Thirunarayanan Balathandayuthapani
7737f15f87 MDEV-31333 fsp_free_page() fails to move the extent from
FSP_FREE_FRAG to FSP_FREE list

- This issue was caused by commit 0b47c126e3.
In fsp_free_page(), InnoDB should set XDES_FREE_BIT of the page before
moving the extent from FSP_FREE_FRAG to FSP_FREE list.
2023-05-24 15:15:29 +05:30
Oleksandr Byelkin
be19f81ad5 Merge branch '10.6' into 10.9 2023-05-24 09:46:27 +02:00
Marko Mäkelä
2cd6a48e72 Merge 10.5 into 10.6 2023-05-24 08:38:35 +03:00
Marko Mäkelä
b220bb756b Merge bb-10.6-release into 10.6 2023-05-24 08:37:19 +03:00
Marko Mäkelä
98de15aba1 Merge bb-10.5-release into bb-10.6-release 2023-05-24 08:36:30 +03:00
Marko Mäkelä
383105dae1 Merge bb-10.5-release into 10.5 2023-05-24 08:28:20 +03:00
Marko Mäkelä
c5cf94b2dc MDEV-31234 fixup: Free some UNDO pages earlier
trx_purge_truncate_rseg_history(): Add a parameter to specify if
the entire rollback segment is safe to be freed. If not, we may
still be able to invoke trx_undo_truncate_start() and free some pages.
2023-05-24 08:25:26 +03:00
Marko Mäkelä
270eeeb523 Merge 10.5 into 10.6 2023-05-23 12:25:39 +03:00
Marko Mäkelä
9c35f9c9c1 MDEV-31234 fixup: Allow innodb_undo_log_truncate=ON after upgrade
trx_purge_truncate_history(): Relax a condition that would prevent
undo log truncation if the undo log tablespaces were "contaminated"
by the bug that commit e0084b9d31 fixed.
That is, trx_purge_truncate_rseg_history() would have invoked
flst_remove() on TRX_RSEG_HISTORY but not reduced TRX_RSEG_HISTORY_SIZE.

To avoid any regression with normal operation, we implement this
fixup during slow shutdown only. The condition on the history list
being empty is necessary: without it, in the test
innodb.undo_truncate_recover there may be much fewer than the
expected 90,000 calls to row_purge() before the truncation.
That is, we would truncate the undo tablespace before actually having
processed all undo log records in it.

To truncate such "contaminated" or "bloated" undo log tablespaces
(when using innodb_undo_tablespaces=2 or more)
you can execute the following SQL:

BEGIN;INSERT mysql.innodb_table_stats VALUES('','',DEFAULT,0,0,0);ROLLBACK;
SET GLOBAL innodb_undo_log_truncate=ON, innodb_fast_shutdown=0;
SHUTDOWN;

The first line creates a dummy InnoDB transaction, to ensure that there
will be some history to be purged during shutdown and that the undo
tablespaces will be truncated.
2023-05-23 12:20:27 +03:00
Monty
a7adfd4c52 Optimized version of safe_strcpy()
Note: We should replace most case of safe_strcpy() with strmake() to avoid
the not needed zerofill.
2023-05-23 10:02:33 +03:00
Monty
92d2ceac73 MDEV-28285 Unexpected result when combining DISTINCT, subselect and LIMIT
The problem was that when  JOIN_TAB::remove_duplicates() noticed there
can only be one possible row in the output, it adjusted limits but
didn't take into account any possible offset.

Fixed by not adjusting limit offset when setting one-row-limit.
2023-05-23 09:16:36 +03:00
Monty
cd37e49422 MDEV-31083 ASAN use-after-poison in myrg_attach_children
The reason for ASAN report was that the MERGE and MYISAM file
had different key definitions, which is not allowed.

Fixed by ensuring that the MERGE code is not copying more key stats
than what is in the MyISAM file.

Other things:
- Give an error if different MyISAM files has different number of
  key parts.
2023-05-23 09:16:36 +03:00
Monty
c7e04af8bc Update main.selectivity test and results 2023-05-23 09:16:36 +03:00