Commit graph

192722 commits

Author SHA1 Message Date
Daniel Black
a35cde8cd8 MDEV-25513: raise systemd LimitNOFILE limits to match server defaults
Quoting MDEV reporter Daniel Lewart:

Starting MariaDB with default configuration causes the following problems:

    "[Warning] Could not increase number of max_open_files to more than 16384 (request: 32186)"
    silently reduces table_open_cache_instances from 8 (default) to 4

Default Server System Variables:

    extra_max_connections = 1
    max_connections = 151
    table_open_cache = 2000
    table_open_cache_instances = 8
    thread_pool_size = 4

LimitNOFILE=16834 is in the following files:

    support-files/mariadb.service.in
    support-files/mariadb@.service.in

Looking at sql/mysqld.cc lines 3837-3917:
wanted_files= (extra_files + max_connections + extra_max_connections +
tc_size * 2 * tc_instances);
wanted_files+= threadpool_size;

Plugging in the default values:
wanted_files = (30 + 151 + 1 + 2000 * 2 * 8 + 4) = 32186

However, systemd configuration has LimitNOFILE = 16384, which is far smaller.

I suggest increasing LimitNOFILE to 32768.
2021-04-27 08:17:40 +10:00
Marko Mäkelä
4d412e9854 MDEV-24758 heap-use-after-poison in innobase_add_instant_try/rec_copy
This is a backport of
commit fd9ca2a742 (MDEV-23295) and
commit 9a156e1a23 (MDEV-23345) to 10.3.

An instant ADD/DROP/reorder column could create a dummy table
object with the wrong ROW_FORMAT when innodb_default_row_format
was changed between CREATE TABLE and ALTER TABLE.

prepare_inplace_alter_table_dict(): If we had promised that
ALGORITHM=INPLACE is supported, we must preserve the ROW_FORMAT.

The rest of the changes are related to adding
Alter_inplace_info::inplace_supported to cache the return value of
handler::check_if_supported_inplace_alter().
2021-04-26 18:17:50 +03:00
Daniel Bartholomew
9db14e93ac
bump the VERSION 2021-04-26 09:51:38 -04:00
Marko Mäkelä
dd07cfcecd MDEV-15756: Remove some garbage output
os_aio_print(): Remove output that should have been removed in
commit 5e62b6a5e0 (MDEV-16264).
2021-04-26 15:30:19 +03:00
Daniele Sciascia
9e6310e323 Fix MTR test wsrep.variables_debug
The test was changing variable wsrep_provider dynamically,
but wsrep_provider was recently made read-only.

followup for ce3a2a688d
2021-04-26 09:56:46 +03:00
Marko Mäkelä
ed4b2b3f95 Merge 10.5 into 10.6 2021-04-26 08:40:36 +03:00
Sujatha
391f1aa6ee MDEV-24773: slave_compressed_protocol doesn't work properly with semi-sync replication
Back port upstream fix

commit 1800b015a1d487330f7b15f2020b887be348a66b
Author: Venkatesh Duggirala <venkatesh.duggirala@oracle.com>
Date:   Fri Sep 8 20:29:22 2017 +0530

Bug#26027024    SLAVE_COMPRESSED_PROTOCOL DOESN'T WORK WITH
SEMI-SYNC REPLICATION IN MYSQL-5.7

Analysis: In mysql-5.6, dump thread (the thread that is created
on Master after Slave requested for a binlog dump) is also used
to receive acknowledgements from the Slave and act on them accordingly.
For performance reasons, a special thread called Ack Receiver thread
is added in mysql-5.7 Semi synchronous replication plugin.
This thread does not have special handling to receive acknowledgements
if Slave has enabled compression in the protocol. Hence Master is
unable to handle any slave if Slave_compressed_protocol is enabled
on it.

Fix: Enable compress flag on the communication channels if the Slave
has Slave_compressed_protocol ON.
2021-04-26 11:09:39 +05:30
Sergei Petrunia
2f6912dabc MDEV-24898: Server crashes in st_select_lex::next_select
(trivial backport to 10.2)
Add a testcase
2021-04-25 21:23:56 +03:00
Sergei Petrunia
c72c77ca3b MDEV-24925: Server crashes in Item_subselect::init_expr_cache_tracker
(trivial backport to 10.2)
The optimizer removes redundant GROUP BY operations. If GROUP BY element
is a subselect, it is "eliminated".

However one must not eliminate the item if it is used both in the select
list and in the GROUP BY, like so:

  select (select ... ) as SUBQ from ... group by SUBQ

Do not eliminate such items.
2021-04-25 21:22:54 +03:00
Marko Mäkelä
b81382887c MDEV-25512 Deadlock between sux_lock::u_x_upgrade() and sux_lock::u_lock()
In the SUX_LOCK_GENERIC implementation, we can remember at most
one pending exclusive lock request. If multiple exclusive lock
requests are pending, the WRITER_WAITING flag will be cleared when
the first waiting writer acquires the exclusive lock.

ssux_lock_low::update_lock(): If WRITER_WAITING is set, wake up
the writer even if the UPDATER flag is set, because the waiting
writer may be in the process of upgrading its U lock to X.

rw_lock::read_unlock(): Also indicate that an X lock waiter must
be woken up if an U lock exists.

This fix may cause unnecessary wake-ups and system calls, but this
is the best that we can do. Ideally we would use the MDEV-25404
idea of a separate 'writer' mutex, but there is no portable way to
request that a non-recursive mutex be created, and InnoDB requires
the ability to transfer buf_block_t::lock ownership to an I/O thread.

To allow problems like this to be caught more reliably in the future,
we add a unit test for srw_mutex, srw_lock, ssux_lock, sux_lock.
2021-04-25 12:58:16 +03:00
Marko Mäkelä
4725792bf3 Merge 10.4 into 10.5 2021-04-25 12:04:45 +03:00
Sergei Petrunia
42aad65b89 MDEV-24898: Server crashes in st_select_lex::next_select
Add a testcase
2021-04-25 10:32:09 +03:00
Sergei Petrunia
393cf51c04 MDEV-24925: Server crashes in Item_subselect::init_expr_cache_tracker
The optimizer removes redundant GROUP BY operations. If GROUP BY element
is a subselect, it is "eliminated".

However one must not eliminate the item if it is used both in the select
list and in the GROUP BY, like so:

  select (select ... ) as SUBQ from ... group by SUBQ

Do not eliminate such items.
2021-04-25 10:32:09 +03:00
Marko Mäkelä
e4394cc547 Merge 10.3 into 10.4 2021-04-25 10:20:57 +03:00
Igor Babaev
2c9bf0ae87 This commit adds the same call of st_select_lex::set_unique_exclude() that
complemented the fix for MDEV-24823 in 10.2. As it is the only call of
this function in 10.3 the commit also has added the code of the function.
2021-04-24 16:53:24 -07:00
Marko Mäkelä
c425d93b92 Merge 10.2 into 10.3
except commit 1288dfffe7
2021-04-24 10:37:21 +03:00
Marko Mäkelä
14a18d7d7f MDEV-23026/MDEV-25474 fixup: Assertion ib_table->stat_initialized
It is possible that an object that was originally created by
open_purge_table() will remain cached and reused for SQL execution.
Our previous fix wrongly assumed that ha_innobase::open() would
always be called before SQL execution starts. Therefore, we must
invoke dict_stats_init() in ha_innobase::info_low() instead of
only doing it in ha_innobase::open().

Note: Concurrent execution of dict_stats_init() on the same table
is possible, but it also was possible between two calls to
ha_innobase::open(), with no ill effects observed.

This should fix the assertion failure on stat_initialized.
A possibly easy way to reproduce it would have been
to run the server with innodb_force_recovery=2 (disable the purge of
history), update a table so that an indexed virtual column will be
affected, and finally restart the server normally (purge enabled),
to observe a crash when the table is accessed from SQL.

The problem was first observed and this fix verified by
Elena Stepanova. Also Thirunarayanan Balathandayuthapani
repeated the problem.
2021-04-24 09:37:46 +03:00
Marko Mäkelä
25ed665a20 MDEV-25459 MVCC read from index on CHAR or VARCHAR wrongly omits rows
row_sel_sec_rec_is_for_clust_rec(): If the field in the
clustered index record stored off page, always fetch it,
also when the secondary index field has been built on the
entire column. This was broken ever since the InnoDB Plugin
for MySQL Server 5.1 introduced ROW_FORMAT=DYNAMIC and
ROW_FORMAT=COMPRESSED for InnoDB tables. That code was first
introduced in this tree in
commit 3945d5e554.

For the original ROW_FORMAT=REDUNDANT and the MySQL 5.0.3
ROW_FORMAT=COMPRESSED, there was no problem, because for
those tables we always stored at least a 768-byte prefix of
each column in the clustered index record.

row_sel_sec_rec_is_for_blob(): Allow prefix_len==0 for matching
the full column.
2021-04-24 09:26:49 +03:00
Igor Babaev
1288dfffe7 This patch complements the patch for MDEV-24823. 2021-04-23 14:32:59 -07:00
Aleksey Midenkov
42f8548ff6 MDEV-25091 CREATE TABLE: field references qualified by a wrong table name succeed
Before FRM is written walk vcol expressions through
check_table_name_processor() and check if field items match (db,
table_name) qualifier.

We cannot do this in check_vcol_func_processor() as there is already
no table name qualifiers in expressions of written and loaded FRM.
2021-04-23 15:20:35 +03:00
Aleksey Midenkov
4649ba7493 MDEV-23455 Hangs + Sig11 in unknown location(s) due to single complex FK query
Buffer overflow in ib_push_warning() fixed by using vsnprintf().

InnoDB parser was obsoleted by MDEV-16417.

Thanks to Nikita Malyavin for review and suggestion.
2021-04-23 14:09:43 +03:00
Sergei Golubchik
018d7440fd remove EXCEPTIONS-CLIENT
It's Oracle libmysqlclient license exception, we no longer include,
build or ship libmysqlclient
2021-04-23 10:38:10 +02:00
Marko Mäkelä
1a647b700f MDEV-25487 Assertion failed in lock_rec_move
row_ins_clust_index_entry_low(): Do not enable bulk insert if
any record locks exist on the table. Bulk insert is assumed to
be covered only by an exclusive table lock, with no row-level
locking or undo logging.
2021-04-23 10:07:08 +03:00
Sujatha
ee20e26dbb MDEV-20220 post push fix
Remove garbage files.
2021-04-23 11:58:39 +05:30
Igor Babaev
e3a25793be MDEV-24823 Crash with invalid multi-table update of view in 2nd execution of SP
Before this patch mergeable derived tables / view used in a multi-table
update / delete were merged before the preparation stage.
When the merge of a derived table / view is performed the on expression
attached to it is fixed and ANDed with the where condition of the select S
containing this derived table / view. It happens after the specification of
the derived table / view has been merged into S. If the ON expression refers
to a non existing field an error is reported and some other mergeable derived
tables / views remain unmerged. It's not a problem if the multi-table
update / delete statement is standalone. Yet if it is used in a stored
procedure the select with incompletely merged derived tables / views may
cause a problem for the second call of the procedure. This does not happen
for select queries using derived tables / views, because in this case their
specifications are merged after the preparation stage at which all ON
expressions are fixed.
This patch makes sure that merging of the derived tables / views used in a
multi-table update / delete statement is performed after the preparation
stage.

Approved by Oleksandr Byelkin <sanja@mariadb.com>
2021-04-22 20:02:08 -07:00
Igor Babaev
b3b5d57e78 MDEV-24823 Crash with invalid multi-table update of view in 2nd execution of SP
Before this patch mergeable derived tables / view used in a multi-table
update / delete were merged before the preparation stage.
When the merge of a derived table / view is performed the on expression
attached to it is fixed and ANDed with the where condition of the select S
containing this derived table / view. It happens after the specification of
the derived table / view has been merged into S. If the ON expression refers
to a non existing field an error is reported and some other mergeable derived
tables / views remain unmerged. It's not a problem if the multi-table
update / delete statement is standalone. Yet if it is used in a stored
procedure the select with incompletely merged derived tables / views may
cause a problem for the second call of the procedure. This does not happen
for select queries using derived tables / views, because in this case their
specifications are merged after the preparation stage at which all ON
expressions are fixed.
This patch makes sure that merging of the derived tables / views used in a
multi-table update / delete statement is performed after the preparation
stage.

Approved by Oleksandr Byelkin <sanja@mariadb.com>
2021-04-22 13:56:50 -07:00
Sergei Golubchik
b8fad8c6bf MDEV-25030 Upgrade to 10.5.9 breaks root's ability to grant 2021-04-22 17:52:08 +02:00
Sergei Golubchik
a40f29ab19 MYSQL_MAINTAINER_MODE fixes
* MSVC check was lost in a bad merge
* add a comment
* move AIX check where it belongs and where it would still
  allow -Werror, if desired
2021-04-22 17:52:08 +02:00
Marko Mäkelä
bfc0110b9b Null-merge the revert of MDEV-24589
While MDEV-24589 causes the regression
MDEV-25491 Race condition between DROP TABLE and purge of
SYS_INDEXES record
we must not revert it from 10.6, because MDEV-24589 is a prerequisite for
MDEV-25180 Atomic ALTER TABLE
that is targeting 10.6. The regression will be dealt with later.
2021-04-22 18:39:40 +03:00
Marko Mäkelä
18f35a8608 Merge 10.5 into 10.6 2021-04-22 18:13:47 +03:00
Vladislav Vaintroub
5c5d24c772 MDEV-25456 - fix predicate in ib::error_or_warn 2021-04-22 16:59:30 +02:00
Vladislav Vaintroub
78bb9533f4 MDEV-25456 MariaBackup logs "[ERROR]" on Invalid log block checksum
Fix is to changed message to be [WARNING] for backup
2021-04-22 15:51:55 +02:00
Vladislav Vaintroub
f6542a7af8 Update timezone data on Windows
There is new Yukon Standard time Windows timezone.

Also fix the powershell script that generates the Windows locale mapping,
tell powershell to use TLSv1.2 to access the github (on some reason it is
TLS1.1 that powershell is using by default, and it does no work)
2021-04-22 15:51:55 +02:00
Marko Mäkelä
8121d03fd4 MDEV-25404 fixup: Fix ssux_lock_low::u_wr_upgrade()
The U-to-X upgrade turned out to be incorrect. A debug assertion
failed in wr_wait(), called from mtr_defer_drop_ahi() in a stress
test with innodb_adaptive_hash_index=ON.

A correct upgrade procedure ought to be readers.fetch_add(WRITER-1)
to register ourselves as a WRITER (or waiting writer) and to release
the reference that was being held for the U lock.

Thanks to Matthias Leich for catching the problem.
2021-04-22 15:43:30 +03:00
Marko Mäkelä
1636db5443 Revert "MDEV-24589 DROP TABLE is not crash-safe"
This reverts commit e731a28394.

A crash occurred during the test stress.ddl_innodb when
fil_delete_tablespace() for DROP TABLE was waiting in
fil_check_pending_operations() and a purge thread for handling
an earlier DROP INDEX was attempting to load the index root page
in btr_free_if_exists() and btr_free_root_check(). The function
buf_page_get_gen() would write out several times
"trying to read...being-dropped tablespace"
before giving up and committing suicide.

It turns out that during any page access in btr_free_if_exists(),
fil_space_t::set_stopping() could have been invoked by
fil_check_pending_operations(), as part of dropping the tablespace.
Preventing this race condition would require extensive changes
to the allocation code or some locking mechanism that would ensure
that we only set the flag if btr_free_if_exists() is not in progress.

Either way, that could be a too risky change in a GA release.
Because MDEV-24589 is not strictly necessary in the 10.5 release
series and it only is a requirement for MDEV-25180 in a later
major release, we will revert the change from 10.5.
2021-04-22 14:57:23 +03:00
sachin
21973d0d8e MDEV-22953 main.flush_read_lock failed in buildbot with XAER_NOTA: Unknown XID
Issue:- Since there is no waiting for the actual disconnection of the con_tmp
(which does XA prepare of test1), We can have a issue when test1 is not
prepared and we are calling rollback on test1 , giving XAER_NOTA: Unknown XID
error

Solution:- Wait for the complete disconnection of con_tmp
2021-04-22 10:37:33 +01:00
Marko Mäkelä
54c460ace6 Merge 10.5 into 10.6 2021-04-22 08:43:03 +03:00
Marko Mäkelä
b728c3dbd9 fixup dd6ad38068: remove dead code 2021-04-22 08:42:55 +03:00
Marko Mäkelä
df33b719ca Merge 10.4 into 10.5 2021-04-22 08:25:40 +03:00
Marko Mäkelä
cbbca7edf6 MDEV-25483: Shutdown crash during innodb.innodb_buffer_pool_resize_temporary
buf_resize_callback(): Correct an invalid assertion, and enable
the assertion in debug builds only.

Between buf_resize_start() and buf_resize_shutdown(),
srv_shutdown_state must be less than SRV_SHUTDOWN_CLEANUP.

The incorrect assertion had been introduced in
commit 5e62b6a5e0 (MDEV-16264).
As a result, the server could crash if shutdown was initiated
concurrently with initiating a change of innodb_buffer_pool_size.
2021-04-22 08:13:57 +03:00
Marko Mäkelä
ee455e6f2e Merge 10.3 into 10.4 2021-04-22 07:51:33 +03:00
Marko Mäkelä
0d267f7caa MDEV-25362 after-merge fix: Remove unnecessary code 2021-04-22 07:36:04 +03:00
Marko Mäkelä
6f271302b6 Merge 10.2 into 10.3 2021-04-22 07:32:51 +03:00
Thirunarayanan Balathandayuthapani
fb96ac0a49 MDEV-25474 Background thread returns uninitialized statistics
to mysql interpreter

InnoDB returns uninitialized statistics to mysql interpreter
when background thread is opening the table. So it leads to
assertion failure. In that case, InnoDB avoid sending
innodb statistics information to mysql interpreter.
2021-04-21 21:33:29 +05:30
Eugene Kosov
64eeb250eb MDEV-25457 Server crashes in row_undo_mod_clust_low upon rollback of read-only transaction
node->index was NULL.
But it's possible to get dict_table_t* from another source.
2021-04-21 17:42:16 +03:00
Andrei Elkin
2a7dd64425 MDEV-24526 binlog rotate via FLUSH LOGS may obsolate binlog file for recovery too eary
There was race between a committing transaction and the following in binlog
order FLUSH LOGS that could create a 2nd Binlog checkpoint (BCP) event
in the new file *before* the first logged-in-old-binlog transaction gets committed in
Innodb. That would cause the transaction loss at recovery, should
the server stop right after the BCP.

The race is tackled by enforcing the necessary set of mutexes to be acquired
by FLUSH-LOGS handler in the correct order (of the group commit leader
pattern).

Note, there remain two cases where a similar race is still possible:
  - the above race as it is when the server is run with ("unlikely")
    non-default `--binlog-optimize-thread-scheduling=0` (MDEV-24530), and
  - at unlikely event of bin-logging of Incident event (MDEV-24531) that
    also triggers binlog rotation,
    in both cases though with lesser chances after the current fixes.
2021-04-21 15:39:32 +03:00
Vicențiu Ciorbaru
bc04ded235 Follow-up post JSON_TABLE merge, table_join_options is not used 2021-04-21 15:17:39 +03:00
Vicențiu Ciorbaru
3019d1cb43 Update columnstore to following cleanup of SELECT_LEX 2021-04-21 14:51:15 +03:00
Vicențiu Ciorbaru
aeccdddedd MDEV-25441 WITH TIES is not respected with SQL_BUFFER_RESULT and constant in ORDER BY
Pushing LIMIT to temp aggregation table is possible, but not when WITH
TIES is used. In a degenerate case with constant ORDER BY, the constant
gets removed and the code assumed the limit is push-able.

Ensure that if WITH TIES is present, that this does not happen.
2021-04-21 14:09:17 +03:00
Vicențiu Ciorbaru
44a6af65f5 MDEV-25430: ROW | ROWS should be a required keyword after OFFSET start
Implemented according to standard OFFSET start { ROW | ROWS},
ROW and ROWS is mandatory after OFFSET.
2021-04-21 14:09:17 +03:00