Commit graph

197012 commits

Author SHA1 Message Date
Sergei Petrunia
2165c30486 Fix testcase for MDEV-31240 to work with --view-protocol. 2023-06-08 11:35:21 +03:00
Sergei Petrunia
b3074128a6 MDEV-31380 post-fix: fix group_min_max.test with embedded and view-protocol
embedded doesn't have optimizer trace,
view-protocol doesn't work with long column names.
2023-06-08 11:14:50 +03:00
Marko Mäkelä
80585c9d6f Merge 10.5 into 10.6 2023-06-08 10:42:56 +03:00
Marko Mäkelä
d3eefbaa55 MDEV-31355 fixup: Adjust one more test
The test gcol.gcol_purge would reliably hang on 10.6 on a
Microsoft Windows builder without this adjustment.
A similar adjustment was applied in
commit 3e40f9a7f3
to the tests innodb.dml_purge and innodb.instant_alter_purge.
2023-06-08 10:40:48 +03:00
Marko Mäkelä
21031b24fc Suppress an occasional buffer pool warning 2023-06-08 09:38:03 +03:00
Marko Mäkelä
6882eeabb0 MDEV-30483 fixup: Declare the test plugin for Debian 2023-06-08 09:23:01 +03:00
Marko Mäkelä
c25b496724 MDEV-31382 SET GLOBAL innodb_undo_log_truncate=ON has no effect on logically empty undo logs
innodb_undo_log_truncate_update(): A callback function. If
SET GLOBAL innodb_undo_log_truncate=ON, invoke
srv_wake_purge_thread_if_not_active().

srv_wake_purge_thread_if_not_active(): If innodb_undo_log_truncate=ON,
always wake up the purge subsystem.

srv_do_purge(): If the history is empty, invoke
trx_purge_truncate_history() in order to free undo log pages.

trx_purge_truncate_history(): If head.trx_no==0, consider the
cached undo logs to be free.

trx_purge(): Remove the parameter "bool truncate" and let the
caller invoke trx_purge_truncate_history() directly.

Reviewed by: Vladislav Lesin
2023-06-08 09:18:21 +03:00
Marko Mäkelä
3e40f9a7f3 MDEV-31355 innodb_undo_log_truncate=ON fails to wait for purge of enough transaction history
purge_sys_t::sees(): Wrapper for view.sees().

trx_purge_truncate_history(): Invoke purge_sys.sees() instead of
comparing to head.trx_no, to determine if undo pages can be safely freed.

The test innodb.cursor-restore-locking was adjusted by Vladislav Lesin,
as was the the debug instrumentation in row_purge_del_mark().

Reviewed by: Vladislav Lesin
2023-06-08 09:17:52 +03:00
Oleksandr Byelkin
04f0b955dd Merge branch '10.6' into 10.6.14 2023-06-07 19:59:52 +02:00
Oleksandr Byelkin
78b1831c9f Merge branch '10.4' into 10.4.30 2023-06-07 15:08:29 +02:00
Marko Mäkelä
609b4e997a MariaDB 10.5.21 release
-----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCgAdFiEEF39AEP5WyjM2MAMF8WVvJMdM0dgFAmSAc6IACgkQ8WVvJMdM
 0dhC5g/9FEU8b3CVSfSRq/VQN2BFYaxABfzHWMQbEnnNhDuNzXzyMz0DuRseGo7B
 dSEsxOrP48TOgnj39Vi+jrR70Fcz8ep6p7VNcMGDrXkj3e2zd9L7ThH5KZ80k8sW
 aLmXjUUzkerNTm89aBKrIN6NvoKDzZEOa/InXtfoXiuU42yR4hqOnn6nfRuD0qEC
 ZLEPLzycSVX/MZPnl2v7WyQhLPWLOHOMgutHwCw03v7RtnhqiDreVhVe/DstJDsZ
 13DcSGLW5DILvQg1HHefT/esPKzp9iPy/HtpAE+dzCGaj96ZlYWregZ0jcA40j8W
 xIeujIk7FVtdWjYIPcBKQch/39iCCgMl1pvqSX6x/Tw2a8ZvrZeNHyZoGISNulCi
 +h0Jp3w3IJr/JJxMCWp7ueXrsYeuESnNyc+6JPNVM4JafxMtoZDxw8wLZyhzB2uO
 88RjMLJ18jEAjTQPYMinxfWbvLSD0/jA+WbQvsR+MNVZuNhD9sG/KsPUpRyQEjUh
 n/MJoa6PeLsuEnHUf9Urj/BWBJa6ZFDaJxHKQzc6gpxiakRVODLd2fGJHZLBcJS7
 W6Sgcj8SKXagkYBLj8pCzpb+FY6gi15rhs4XA3A/rlUbEYRaERgW05//6OF7Ghwn
 IuTv9KF/O8gVrJ4Xn2+f0HJh1Ssj4EI76R7J+zWfcNcScSxATSE=
 =C0PS
 -----END PGP SIGNATURE-----

Merge mariadb-10.5.21 into 10.5
2023-06-07 15:31:55 +03:00
Daniel Bartholomew
a77939ec8c
bump the VERSION 2023-06-07 08:12:43 -04:00
Daniel Bartholomew
91367e82f1
bump the VERSION 2023-06-07 08:10:48 -04:00
Daniel Bartholomew
78a1f3ce81
bump the VERSION 2023-06-07 08:09:02 -04:00
Sergei Petrunia
a0e7bd735b MDEV-31380: Assertion `s->table->opt_range_condition_rows <= s->found_records' failed
LooseScan code set opt_range_condition_rows to be the

  MIN(loose_scan_plan->records, table->records)

totally ignoring possible quick range selects.  If there was a quick
select $QUICK on another index with

  $QUICK->records < loose_scan_plan->records

this would create a situation where

   opt_range_condition_rows > $QUICK->records

which causes an assert in 10.6+ and potentially wrong query plan
choice in 10.5.

Fixed by making opt_range_condition_rows to be the minimum #rows
of any quick select.

Approved-by: Monty <monty@mariadb.org>
2023-06-07 13:54:34 +03:00
Yuchen Pei
7e17a88e75
MDEV-30435 MDEV-30981 Fix ubsan errors w.r.t. memcpy in spd_trx.cc
Extract the indexed string memcopy pattern in spd_trx.cc to a static
inline function.

Also updated the ubsan check in mdev_26541.test (h/t roel).
2023-06-07 14:07:31 +10:00
Brandon Nesterenko
8ed88e3455 Revert "MDEV-13915: STOP SLAVE takes very long time on a busy system"
This reverts commit 0a99d457b3
because it should go into only 10.5+
2023-06-06 08:11:38 -06:00
Sergei Golubchik
677d6f0f23 MDEV-31183 binlog_encryption.encrypted_master_switch_to_unencrypted_gtid fails in BB with UBSAN runtime error: downcast of address
sql/log.cc:11101:56: runtime error: downcast of address 0x7f9dc801e9c8 which does not point to an object of type 'Gtid_list_log_event'
sql/sql_repl.cc:1429:12: runtime error: member call on address 0x7f1ca401ea48 which does not point to an object of type 'Gtid_list_log_event'
2023-06-06 13:22:39 +02:00
Marko Mäkelä
c93754d45e MDEV-31234 related cleanup
trx_purge_free_segment(), trx_purge_truncate_rseg_history():
Replace some unreachable code with debug assertions.
A buffer-fix does prevent pages from being evicted
from the buffer pool; see buf_page_t::can_relocate().

Tested by: Matthias Leich
2023-06-05 18:53:20 +02:00
Sergei Golubchik
a42a6fa99b Merge branch 'bb-10.5-release' into bb-10.6-release 2023-06-05 18:53:02 +02:00
Brandon Nesterenko
0a99d457b3 MDEV-13915: STOP SLAVE takes very long time on a busy system
The problem is that a parallel replica would not immediately stop
running/queued transactions when issued STOP SLAVE. That is, it
allowed the current group of transactions to run, and sometimes the
transactions which belong to the next group could be started and run
through commit after STOP SLAVE was issued too, if the last group
had started committing. This would lead to long periods to wait for
all waiting transactions to finish.

This patch updates a parallel replica to try and abort immediately
and roll-back any ongoing transactions. The exception to this is any
transactions which are non-transactional (e.g. those modifying
sequences or non-transactional tables), and any prior transactions,
will be run to completion.

The specifics are as follows:

 1. A new stage was added to SHOW PROCESSLIST output for the SQL
Thread when it is waiting for a replica thread to either rollback or
finish its transaction before stopping. This stage presents as
“Waiting for worker thread to stop”

 2. Worker threads which error or are killed no longer perform GCO
cleanup if there is a concurrently running prior transaction. This
is because a worker thread scheduled to run in a future GCO could be
killed and incorrectly perform cleanup of the active GCO.

 3. Refined cases when the FL_TRANSACTIONAL flag is added to GTID
binlog events to disallow adding it to transactions which modify
both transactional and non-transactional engines when the binlogging
configuration allow the modifications to exist in the same event,
i.e. when using binlog_direct_non_trans_update == 0 and
binlog_format == statement.

 4. A few existing MTR tests relied on the completion of certain
transactions after issuing STOP SLAVE, and were re-recorded
(potentially with added synchronizations) under the new rollback
behavior.

Reviewed By
===========
Andrei Elkin <andrei.elkin@mariadb.com>
2023-06-05 10:03:06 -06:00
Sergei Golubchik
bed70468ea Merge branch 'bb-10.4-release' into bb-10.5-release 2023-06-05 17:50:51 +02:00
Sergei Petrunia
928012a27a MDEV-31403: Server crashes in st_join_table::choose_best_splitting
The code in choose_best_splitting() assumed that the join prefix is
in join->positions[].

This is not necessarily the case. This function might be called when
the join prefix is in join->best_positions[], too.
Follow the approach from best_access_path(), which calls this function:
pass the current join prefix as an argument,
"const POSITION *join_positions" and use that.
2023-06-05 18:24:39 +03:00
heyingquan0030
8de6740a2f MDEV-31205 Typo: complatible > compatible 2023-06-05 09:16:44 +10:00
Otto Kekalainen
fcfd6361cb Deb: Fix blocksize check to use $mysql_datadir/$datadir correctly
In commit f99a8918 this line was changed to not use awk, and new version
copied both to init file and preinst file but overlooking that they use
different variable names.

Also fix minor syntax issues to make Shellcheck happy.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.
2023-06-03 11:12:37 +02:00
Marko Mäkelä
cf37e44eec MDEV-31350: Hang in innodb.recovery_memory
buf_flush_page_cleaner(): Whenever buf_pool.ran_out(), invoke
buf_pool.get_oldest_modification(0) so that all clean blocks
will be removed from buf_pool.flush_list and buf_flush_LRU_list_batch()
will be able to evict some pages.

This fixes a regression that was likely caused by
commit a55b951e60 (MDEV-26827).
2023-06-03 11:12:32 +02:00
Marko Mäkelä
dd298873da MDEV-31309 Innodb_buffer_pool_read_requests is not updated correctly
srv_export_innodb_status(): Update
export_vars.innodb_buffer_pool_read_requests as it was done
before commit a55b951e60 (MDEV-26827).
If innodb_status_variables[] pointed to a sharded variable, it would
only access the first shard.
2023-06-03 11:12:27 +02:00
Marko Mäkelä
89eb6fa8a7 MDEV-31308 InnoDB monitor trx_rseg_history_len was accidentally disabled by default
innodb_counter_info[]: Revert a change that was accidentally made in
commit 204e7225dc
2023-06-03 11:12:21 +02:00
Marko Mäkelä
883333a74e MDEV-31158: Potential hang with ROW_FORMAT=COMPRESSED tables
btr_cur_need_opposite_intention(): Check also page_zip_available()
so that we will escalate to exclusive index latch when a non-leaf
page may have to be split further due to ROW_FORMAT=COMPRESSED page
overflow.

Tested by: Matthias Leich
2023-06-03 11:12:16 +02:00
Marko Mäkelä
459eb9a686 MDEV-29593 fixup: Avoid a leak if rseg.undo_cached is corrupted
trx_purge_truncate_rseg_history(): Avoid a leak similar to the one
that was fixed in MDEV-31324, in case a supposedly cached undo log
page is not found in the rseg.undo_cached list.
2023-06-03 11:12:11 +02:00
Marko Mäkelä
e89bd39c9b MDEV-31343 Another server hang with innodb_undo_log_truncate=ON
trx_purge_truncate_history(): While waiting for a write-fixed block
to become available, simply wait for an exclusive latch on it.
Also, simplify the iteration: first check for oldest_modification>2
(to ignore clean pages or pages belonging to the temporary tablespace)
and then compare the tablespace identifier.

Before releasing buf_pool.flush_list_mutex we will buffer-fix the block
of interest. In that way, buf_page_t::can_relocate() will not hold on
the block and it must remain in the buffer pool until we have acquired
an exclusive latch on it. If the block is still dirty, we will register
it with the tablespace truncation mini-transaction; else, we will simply
release the latch and buffer-fix and move to the next block.

This also reverts commit c4d7939989
because that fix should no longer be necessary; the wait for an
exclusive block latch should allow buf_pool_t::release_freed_page()
on the same block to proceed.

Tested by: Axel Schwenke, Matthias Leich
2023-06-03 11:12:03 +02:00
Marko Mäkelä
3b4b512d8e MDEV-31234 fixup: Allow innodb_undo_log_truncate=ON after upgrade
trx_purge_truncate_history(): Relax a condition that would prevent
undo log truncation if the undo log tablespaces were "contaminated"
by the bug that commit e0084b9d31 fixed.
That is, trx_purge_truncate_rseg_history() would have invoked
flst_remove() on TRX_RSEG_HISTORY but not reduced TRX_RSEG_HISTORY_SIZE.

To avoid any regression with normal operation, we implement this
fixup during slow shutdown only. The condition on the history list
being empty is necessary: without it, in the test
innodb.undo_truncate_recover there may be much fewer than the
expected 90,000 calls to row_purge() before the truncation.
That is, we would truncate the undo tablespace before actually having
processed all undo log records in it.

To truncate such "contaminated" or "bloated" undo log tablespaces
(when using innodb_undo_tablespaces=2 or more)
you can execute the following SQL:

BEGIN;INSERT mysql.innodb_table_stats VALUES('','',DEFAULT,0,0,0);ROLLBACK;
SET GLOBAL innodb_undo_log_truncate=ON, innodb_fast_shutdown=0;
SHUTDOWN;

The first line creates a dummy InnoDB transaction, to ensure that there
will be some history to be purged during shutdown and that the undo
tablespaces will be truncated.
2023-06-03 10:59:53 +02:00
Marko Mäkelä
48d6a5f61b MDEV-31234 fixup: Free some UNDO pages earlier
trx_purge_truncate_rseg_history(): Add a parameter to specify if
the entire rollback segment is safe to be freed. If not, we may
still be able to invoke trx_undo_truncate_start() and free some pages.
2023-06-03 10:59:47 +02:00
Marko Mäkelä
318012a80a MDEV-31234 InnoDB does not free UNDO after the fix of MDEV-30671
trx_purge_truncate_history(): Only call trx_purge_truncate_rseg_history()
if the rollback segment is safe to process. This will avoid leaking undo
log pages that are not yet ready to be processed. This fixes a regression
that was introduced in
commit 0de3be8cfd (MDEV-30671).

trx_sys_t::any_active_transactions(): Separately count XA PREPARE
transactions.

srv_purge_should_exit(): Terminate slow shutdown if the history size
does not change and XA PREPARE transactions exist in the system.
This will avoid a hang of the test innodb.recovery_shutdown.

Tested by: Matthias Leich
2023-06-03 10:59:42 +02:00
Sergei Golubchik
eb472f77e3 Revert "MDEV-30473 : Do not allow GET_LOCK() / RELEASE_LOCK() in cluster"
This reverts commit b05218e08f.
2023-06-03 10:39:34 +02:00
Sergei Golubchik
0fd54c9892 Revert "MDEV-30473 : Do not allow GET_LOCK() / RELEASE_LOCK() in cluster"
This reverts commit 844ddb1109.

This fixes MDEV-30967, MDEV-31325, MDEV-31388
2023-06-03 10:39:34 +02:00
Igor Babaev
8f3bf593d2 MDEV-31240 Crash with condition pushable into derived and containing outer reference
This bug could affect queries containing a subquery over splittable derived
tables and having an outer references in its WHERE clause. If such subquery
contained an equality condition whose left part was a reference to a column
of the derived table and the right part referred only to outer columns
then the server crashed in the function st_join_table::choose_best_splitting()
The crashing code was added in the commit ce7ffe61d8
that made the code of the function sensitive to presence of the flag
OUTER_REF_TABLE_BIT in the KEYUSE_EXT::needed_in_prefix fields.

The field needed_in_prefix of the KEYUSE_EXT structure should not contain
table maps with OUTER_REF_TABLE_BIT or RAND_TABLE_BIT.

Note that this fix is quite conservative: for affected queries it just
returns the query plans that were used before the above mentioned commit.
In fact the equalities causing crashes should be pushed into derived tables
without any usage of split optimization.

Approved by Sergei Petrunia <sergey@mariadb.com>
2023-06-03 10:39:34 +02:00
Igor Babaev
aa713f5ae2 MDEV-31224 Crash with EXPLAIN EXTENDED for multi-table update of system table
EXPLAIN EXTENDED should always print the field item used in the left part
of an equality expression from the SET clause of an update statement as a
reference to table column.

Approved by Oleksandr Byelkin <sanja@mariadb.com>
2023-06-03 10:39:34 +02:00
Daniel Bartholomew
54324e542f bump the VERSION 2023-06-03 10:39:34 +02:00
Sergei Golubchik
c05ecda61f fix string literal escaping in views
process multibyte characters correctly, don't escape half of the character
2023-06-02 17:51:40 +02:00
Sergei Golubchik
69684f689c use correct collation_connection in --view
mysqltest should use the same collation_connection in the service
connection (that creates views) as in the main connection

this makes weight_string("aaa") to return the expected value in --view
and fixes main.func_str failure in --view
2023-06-02 17:51:40 +02:00
Sergei Golubchik
c0463704c2 fix the test for --view 2023-06-02 17:51:40 +02:00
Sergei Golubchik
aca641da28 mtr: handle the case of existing but unreadable /proc/cpuinfo 2023-06-02 17:51:40 +02:00
Sergei Golubchik
d14c485e1c test fixes for 32bit
* disable main.join_cache_notasan on 32bit
  as it uses join_buffer_size=5250229460064350213;

* update sysvars_server_embedded,32bit.rdiff
2023-06-02 17:51:02 +02:00
Sergei Golubchik
d785fa8d0b cmake warnings 2023-06-02 12:53:25 +02:00
Sergei Golubchik
270c233847 clarify why cmake is looking for Java and JNI 2023-06-02 10:46:13 +02:00
Marko Mäkelä
f569e06e03 MDEV-31385 Change buffer stale entries leads to corruption while reusing page
buf_page_free(): If buffered changes existed for the page, drop them.

Co-developed with Thirunarayanan Balathandayuthapani
2023-06-02 11:06:09 +03:00
Marko Mäkelä
8a86df37ef MDEV-31088 Server freeze due to innodb_change_buffering
A 3-thread deadlock has been frequently observed when using
innodb_change_buffering!=none and innodb_file_per_table=0:

(1) ibuf_merge_or_delete_for_page() holding an exclusive latch on the block
and waiting for an exclusive tablespace latch in fseg_page_is_allocated()
(2) btr_free_but_not_root() in fseg_free_step() waiting for an
exclusive tablespace latch
(3) fsp_alloc_free_page() holding the exclusive tablespace latch and waiting
for a latch on the block, which it is reallocating for something else

While this was reproduced using innodb_file_per_table=0, this hang should
be theoretically possible in .ibd files as well, when the recovery or
cleanup of a failed DROP INDEX or ADD INDEX is executing concurrently
with something that involves page allocation.

ibuf_merge_or_delete_for_page(): Avoid invoking fseg_page_is_allocated()
when block==nullptr. The call was redundant in this case, and it could
cause deadlocks due to latching order violation.

ibuf_read_merge_pages(): Acquire an exclusive tablespace latch
before invoking buf_page_get_gen(), which may cause
fseg_page_is_allocated() to be invoked in ibuf_merge_or_delete_for_page().

Note: This will not fix all latching order violations in this area!
Deadlocks involving ibuf_merge_or_delete_for_page(block!=nullptr) are
still possible if the caller is not acquiring an exclusive tablespace latch
upfront. This would be the case in any read operation that involves a
change buffer merge, such as SELECT, CHECK TABLE, or any DML operation that
cannot be buffered in the change buffer.
2023-06-02 10:44:34 +03:00
Sergei Golubchik
dc9498beb6 Revert "MDEV-31230: Fix CONNECT_JDBC in CMake"
This reverts commit 1d0e3d80d8.
2023-06-02 09:02:41 +02:00
Yuchen Pei
bd1eb89d7f Adding .ccls-cache/ to .gitignore
generated by the language server ccls
2023-06-02 11:46:14 +10:00