Commit graph

191,256 commits

Author SHA1 Message Date
Marko Mäkelä
cc5f4428b8 MDEV-23693 fixup: Remove unused btr_search_t::withdraw_clock 2020-10-28 08:13:06 +02:00
Marko Mäkelä
527ade2590 MDEV-23163 Merge new release of InnoDB 5.7.32 to 10.2
All relevant InnoDB changes from MySQL 5.7.32 have been applied
in preceding commits.
2020-10-28 07:27:18 +02:00
Varun Gupta
db56f9b852 MDEV-24015: SQL Error (1038): Out of sort memory when enough memory for the sort buffer is provided
For a correlated subquery filesort is executed multiple times.
During each execution, sortlength() computed total sort key length in
Sort_keys::sort_length, without resetting it first.

Eventually Sort_keys::sort_length got larger than @@sort_buffer_size, which
caused filesort() to be aborted with error.

Fixed by making sortlength() to compute lengths only during the first
invocation. Subsequent invocations return pre-computed values.
2020-10-28 10:53:22 +05:30
Daniele Sciascia
46c273892e MDEV-23623 - Fix assertion in MTR test galera_sr.GCF-1051
Fix assertion `thd->in_active_multi_stmt_transaction() ||
thd->m_transaction_psi == __null' failed on MTR test
galera_sr.GCF-1051.

Add a new MTR test MDEV-23623 that reproduces the issue
deterministically and update wsrep-lib submodule, containing
the actual fix.

Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>
2020-10-28 06:49:56 +02:00
Vladislav Vaintroub
3829b408d6 MDEV-24040 Named pipe permission issue
Tighten access control - deny FILE_CREATE_PIPE_INSTANCE permission to
everyone except current user (the one that runs mysqld)
2020-10-27 22:39:21 +01:00
Eugene Kosov
afc9d00c66 MDEV-23991 dict_table_stats_lock() has unnecessarily long scope
Patch removes dict_index_t::stats_latch. Table/index statistics now
protected with dict_sys->mutex. That way statistics computation can
happen in parallel in several threads and dict_sys->mutex will be locked
only for a short period of time.

This patch is a joint work with Marko Mäkelä

dict_index_t:🔒 make mutable which allows to pass const pointer
when only lock is touched in an object

btr_height_get()
btr_get_size(): make index argument const for better type safety

btr_estimate_number_of_different_key_vals(): now returns computed values
instead of setting fields in dict_index_t directly

remove everything related to dict_index_t::stats_latch

dict_stats_index_set_n_diff(): now returns computed values instead
of setting fields in dict_index_t directly

dict_stats_analyze_index():  now returns computed values instead
of setting fields in dict_index_t directly

Reviewed by: Marko Mäkelä
2020-10-27 19:09:20 +03:00
Sergei Golubchik
2cec0523eb INET6 type plugin -> Beta 2020-10-27 16:45:35 +01:00
Anel Husakovic
e183aec1d7 MDEV-24018: SIGSEGV in Item_func_nextval::update_table on SELECT SETVAL
Reviewed-by: wlad@mariadb.com
2020-10-27 15:17:54 +01:00
Marko Mäkelä
42e1815ad8 MDEV-16952 Introduce SET GLOBAL innodb_max_purge_lag_wait
Let us introduce a dummy variable innodb_max_purge_lag_wait
for waiting that the InnoDB history list length is below
the user-specified limit. Specifically,

SET GLOBAL innodb_max_purge_lag_wait=0;

should wait for all history to be purged. This could be useful
when upgrading from an older version to MariaDB 10.3 or later,
to avoid hitting MDEV-15912.

Note: the history cannot be purged if there exist transactions
that may see old versions.

Reviewed by: Vladislav Vaintroub
2020-10-27 15:47:18 +02:00
Alexey Botchkov
8761571a71 MDEV-22524 SIGABRT in safe_mutex_unlock with
session_track_system_variables and max_relay_log_size.

lock LOCK_global_system_variables around the get_one_variable() call
in the Session_sysvars_tracker::store_variable().
2020-10-27 16:44:11 +04:00
Thirunarayanan Balathandayuthapani
bc540b8706 MDEV-23693 Failing assertion: my_atomic_load32_explicit(&lock->lock_word, MY_MEMORY_ORDER_RELAXED) == X_LOCK_DECR
InnoDB frees the block lock during buffer pool shrinking when other
thread is yet to release the block lock.  While shrinking the
buffer pool, InnoDB allows the page to be freed unless it is buffer
fixed. In some cases, InnoDB releases the latch after unfixing the
block.

Fix:
====
- InnoDB should unfix the block after releases the latch.

- Add more assertion to check buffer fix while accessing the page.

- Introduced block_hint structure to store buf_block_t pointer
and allow accessing the buf_block_t pointer only by passing a
functor. It returns original buf_block_t* pointer if it is valid
or nullptr if the pointer become stale.

- Replace buf_block_is_uncompressed() with
buf_pool_t::is_block_pointer()

This change is motivated by a change in mysql-5.7.32:
mysql/mysql-server@46e60de444
Bug #31036301 ASSERTION FAILURE: SYNC0RW.IC:429:LOCK->LOCK_WORD
2020-10-27 18:30:00 +05:30
Dmitry Shulga
97b10b7fdc MDEV-22805: SIGSEGV in check_fields on UPDATE
For debug build of MariaDB server running of the following test case
will hit the assert `thd->lex->sql_command == SQLCOM_UPDATE' in the function
check_fields() on attempt to execute the UPDATE statement.

  CREATE TABLE t1 (a INT);
  UPDATE t1 FOR PORTION OF APPTIME FROM (SELECT 1 FROM t1) TO 2 SET a = 1;

Stack trace to the fired assert statement
  DBUG_ASSERT(thd->lex->sql_command == SQLCOM_UPDATE)
listed below:
  mysql_execute_command() ->
    mysql_multi_update_prepare() -->
      Multiupdate_prelocking_strategy::handle_end() -->
        check_fiels()

It's worth to note that this stack trace looks like a multi update
statement is being executed. The fired assert is checked inside the
function check_fields() in case table->has_period() returns the value
true that in turns happens when temporal period specified in the UPDATE
statement. Condition specified in the DEBUG_ASSERT statement returns
the false value since the data member thd->lex->sql_command have the
value SQLCOM_UPDATE_MULTI. So, the main question is why a program control
flow go to the path prescribed for handling MULTI update statement
despite of the fact that the ordinary UPDATE statement being executed.

The answer is a way that SQL grammar rules written.

When the statement
  UPDATE t1 FOR PORTION OF APPTIME FROM (SELECT 1 FROM t1) TO 2 SET a = 1;
being parsed an action for the rule 'table_primary_ident' (part of this action
is listed below to simplify description) is  invoked to handle the table
name 't1' specified in the clause 'SELECT 1 FROM t1'.

table_primary_ident:
  table_ident opt_use_partition opt_for_system_time_clause
  opt_table_alias_clause opt_key_definition
  {
    SELECT_LEX *sel= Select;
    sel->table_join_options= 0;
    if (!($$= Select->add_table_to_list(thd, $1, $4,

This action calls the method st_select_lex::add_table_to_list()
to add the table name 't1' to the list of tables being used by the statement.

Later, an action for the following grammar rule
update_table_list:
  table_ident opt_use_partition for_portion_of_time_clause
  opt_table_alias_clause opt_key_definition
  {
    SELECT_LEX *sel= Select;
    sel->table_join_options= 0;
    if (!($$= Select->add_table_to_list(thd, $1, $4,

is invoked to handle the clause 't1 FOR PORTION OF APPTIME FROM ... TO 2'.
This action also calls the method st_select_lex::add_table_to_list()
to add the table name 't1' to the list of tables being used by the statement.

In result the table name 't1' contained twice in this list.

Presence of duplicate names for the table 't1' in a list of table used by
a statement leads to the fact that the function unique_table() called
from the function mysql_update() returns the value true that forces
implementation of the function mysql_update() to return the value 2 as
a signal to fall through the case boundary of the switch statement placed
in the function mysql_execute_statement() and start handling of the case
for sql_command SQLCOM_UPDATE_MULTI. The compound statement block for the
case SQLCOM_UPDATE_MULTI invokes the function mysql_multi_update_prepare()
that executes the statement
  set thd->lex->sql_command= SQLCOM_UPDATE_MULTI;
and after that calls the method
  Multiupdate_prelocking_strategy::handle_end(). Finally, this method
invokes the check_field() function and assert is fired.

The above analysis shows that update for a table that simultaneously specified
both as a destination table of UPDATE statement and as a table taking part in
subquery is actually treated by MariaDB server as multi-update statement.
Taking into account that multi-update statement for temporal period
table is not supported yet by MariaDB, correct way to fix the bug is to return
the error ER_NOT_SUPPORTED_YET for this case.
2020-10-27 18:55:22 +07:00
mkaruza
6a614d6934 MDEV-22707: galera got stuck after flush tables
Deadlock is possible between applier thread and local committing thread with active FLUSH TABLE.
Applier thread should skip table share checks and locks when opening table.

Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>
2020-10-27 11:28:45 +02:00
Sergei Golubchik
d03ea82759 test case for BUG#31650096 2020-10-27 09:24:15 +01:00
Sergei Golubchik
a7d5e85c49 cleanup: have_static_innodb.inc
and remove unused files
2020-10-27 09:24:15 +01:00
Karthik Kamath
1269fd420d BUG#31650096: MYSQL SERVER HEAP-USE-AFTER-FREE IN TRANS_SAVEPOINT
ANALYSIS:
=========
During Bootstrap, while executing the statements from sql
file passed to the init-file server option, transaction
mem_root was being freed for every statement. This creates
an issue with multi statement transactions especially when a
statement in the transaction has to access the memory used
by the previous statement in the transaction.

FIX:
====
Transaction mem_root is freed whenever a transaction is
committed or rolled-back. Hence explicitly freeing it is not
necessary in the bootstrap implementation.

Change-Id: I40f71d49781bf7ad32d474bb176bd6060c9377dc
2020-10-27 09:24:15 +01:00
Sergei Golubchik
0c3723e1d5 Bug#31304432 "INSUFFICIENT PRIVILEGE CHECK BY LOCK TABLES"
`LOCK TABLES view_name` should require
* invoker to have SELECT and LOCK TABLES privileges on the view
* either invoker or definer (only if sql security definer) to
  have SELECT and LOCK TABLES privileges on the used tables/views.
2020-10-27 09:24:15 +01:00
Sergei Golubchik
320a73f6a2 cleanup: PRIV_LOCK_TABLES (10.5 style) 2020-10-27 09:24:14 +01:00
Marko Mäkelä
00ddea4f2f MDEV-24024 innodb.ibuf_not_empty failed in buildbot
Probably due to the changes to page flushing in MDEV-23399
(commit 7cffb5f6e8) the command
CHECK TABLE would occasionally report a different number of rows
for the corrupted secondary index. (The reported number was 991
instead of 990 on one occasion.)

Let us map all numbers to 990 in the output. We only care that the
injected corruption will be detected.
2020-10-27 09:52:42 +02:00
Vicențiu Ciorbaru
8584349108 MDEV-14945 possible buffer overflow in stack resolver
According to https://stackoverflow.com/questions/22827510/how-to-avoid-bad-fd-set-buffer-overflow-crash
it seems that using select instead of poll can cause additional memory
allocations. As we are in a crashed state, we must prevent allocating
any memory (if possible). Thus, switch select call to poll.

Also move some bigger datastructures to global space. The code is not
run in a multithreaded context so best we don't use up stack space
if it's not needed.
2020-10-26 21:02:13 +02:00
Marko Mäkelä
c27e53f459 MDEV-23855: Use normal mutex for log_sys.mutex, log_sys.flush_order_mutex
With an unreasonably small innodb_log_file_size, the page cleaner
thread would frequently acquire log_sys.flush_order_mutex and spend
a significant portion of CPU time spinning on that mutex when
determining the checkpoint LSN.
2020-10-26 17:53:55 +02:00
Marko Mäkelä
a5a2ef079c MDEV-23855: Implement asynchronous doublewrite
Synchronous writes and calls to fdatasync(), fsync() or
FlushFileBuffers() would ruin performance. So, let us
submit asynchronous writes for the doublewrite buffer.
We submit a single request for the likely case that the
two doublewrite buffers are contiquous in the system tablespace.

buf_dblwr_t::flush_buffered_writes_completed(): The completion callback
of buf_dblwr_t::flush_buffered_writes().

os_aio_wait_until_no_pending_writes(): Also wait for doublewrite batches.

buf_dblwr_t::element::space: Remove. We can simply use
element::request.node->space instead.

Reviewed by: Vladislav Vaintroub
2020-10-26 17:53:55 +02:00
Marko Mäkelä
ef3f71fa74 MDEV-23399 fixup: Interleaved doublewrite batches
Author: Vladislav Vaintroub
2020-10-26 17:53:54 +02:00
Marko Mäkelä
8cb01c51fb MDEV-16264 fixup: Clean up asynchronous I/O
os_aio_userdata_t: Remove. It was basically duplicating IORequest.

buf_page_write_complete(): Take only IORequest as a parameter.

os_aio_func(), pfs_os_aio_func(): Replaced with os_aio() that has
no redundant parameters. There is only one caller, so there is no
point to pass __FILE__, __LINE__ as a parameter.
2020-10-26 17:53:54 +02:00
Marko Mäkelä
118e258aaa MDEV-23855: Shrink fil_space_t
Merge n_pending_ios, n_pending_ops to std::atomic<uint32_t> n_pending.
Change some more fil_space_t members to uint32_t to reduce
the memory footprint.

fil_space_t::add(), fil_ibd_create(): Attach the already opened
handle to the tablespace, and enforce the fil_system.n_open limit.

dict_boot(): Initialize fil_system.max_assigned_id.

srv_boot(): Call srv_thread_pool_init() before anything else,
so that files should be opened in the correct mode on Windows.

fil_ibd_create(): Create the file in OS_FILE_AIO mode, just like
fil_node_open_file_low() does it.

dict_table_t::is_accessible(): Replaces fil_table_accessible().

Reviewed by: Vladislav Vaintroub
2020-10-26 17:53:54 +02:00
Marko Mäkelä
45ed9dd957 MDEV-23855: Remove fil_system.LRU and reduce fil_system.mutex contention
Also fixes MDEV-23929: innodb_flush_neighbors is not being ignored
for system tablespace on SSD

When the maximum configured number of file is exceeded, InnoDB will
close data files. We used to maintain a fil_system.LRU list and
a counter fil_node_t::n_pending to achieve this, at the huge cost
of multiple fil_system.mutex operations per I/O operation.

fil_node_open_file_low(): Implement a FIFO replacement policy:
The last opened file will be moved to the end of fil_system.space_list,
and files will be closed from the start of the list. However, we will
not move tablespaces in fil_system.space_list while
i_s_tablespaces_encryption_fill_table() is executing
(producing output for INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION)
because it may cause information of some tablespaces to go missing.
We also avoid this in mariabackup --backup because datafiles_iter_next()
assumes that the ordering is not changed.

IORequest: Fold more parameters to IORequest::type.

fil_space_t::io(): Replaces fil_io().

fil_space_t::flush(): Replaces fil_flush().

OS_AIO_IBUF: Remove. We will always issue synchronous reads of the
change buffer pages in buf_read_page_low().

We will always ignore some errors for background reads.

This should reduce fil_system.mutex contention a little.

fil_node_t::complete_write(): Replaces fil_node_t::complete_io().
On both read and write completion, fil_space_t::release_for_io()
will have to be called.

fil_space_t::io(): Do not acquire fil_system.mutex in the normal
code path.

xb_delta_open_matching_space(): Do not try to open the system tablespace
which was already opened. This fixes a file sharing violation in
mariabackup --prepare --incremental.

Reviewed by: Vladislav Vaintroub
2020-10-26 17:09:01 +02:00
Marko Mäkelä
3a9a3be1c6 MDEV-23855: Improve InnoDB log checkpoint performance
After MDEV-15053, MDEV-22871, MDEV-23399 shifted the scalability
bottleneck, log checkpoints became a new bottleneck.

If innodb_io_capacity is set low or innodb_max_dirty_pct_lwm is
set high and the workload fits in the buffer pool, the page cleaner
thread will perform very little flushing. When we reach the capacity
of the circular redo log file ib_logfile0 and must initiate a checkpoint,
some 'furious flushing' will be necessary. (If innodb_flush_sync=OFF,
then flushing would continue at the innodb_io_capacity rate, and
writers would be throttled.)

We have the best chance of advancing the checkpoint LSN immediately
after a page flush batch has been completed. Hence, it is best to
perform checkpoints after every batch in the page cleaner thread,
attempting to run once per second.

By initiating high-priority flushing in the page cleaner as early
as possible, we aim to make the throughput more stable.

The function buf_flush_wait_flushed() used to sleep for 10ms, hoping
that the page cleaner thread would do something during that time.
The observed end result was that a large number of threads that call
log_free_check() would end up sleeping while nothing useful is happening.

We will revise the design so that in the default innodb_flush_sync=ON
mode, buf_flush_wait_flushed() will wake up the page cleaner thread
to perform the necessary flushing, and it will wait for a signal from
the page cleaner thread.

If innodb_io_capacity is set to a low value (causing the page cleaner to
throttle its work), a write workload would initially perform well, until
the capacity of the circular ib_logfile0 is reached and log_free_check()
will trigger checkpoints. At that point, the extra waiting in
buf_flush_wait_flushed() will start reducing throughput.

The page cleaner thread will also initiate log checkpoints after each
buf_flush_lists() call, because that is the best point of time for
the checkpoint LSN to advance by the maximum amount.

Even in 'furious flushing' mode we invoke buf_flush_lists() with
innodb_io_capacity_max pages at a time, and at the start of each
batch (in the log_flush() callback function that runs in a separate
task) we will invoke os_aio_wait_until_no_pending_writes(). This
tweak allows the checkpoint to advance in smaller steps and
significantly reduces the maximum latency. On an Intel Optane 960
NVMe SSD on Linux, it reduced from 4.6 seconds to 74 milliseconds.
On Microsoft Windows with a slower SSD, it reduced from more than
180 seconds to 0.6 seconds.

We will make innodb_adaptive_flushing=OFF simply flush innodb_io_capacity
per second whenever the dirty proportion of buffer pool pages exceeds
innodb_max_dirty_pages_pct_lwm. For innodb_adaptive_flushing=ON we try
to make page_cleaner_flush_pages_recommendation() more consistent and
predictable: if we are below innodb_adaptive_flushing_lwm, let us flush
pages according to the return value of af_get_pct_for_dirty().

innodb_max_dirty_pages_pct_lwm: Revert the change of the default value
that was made in MDEV-23399. The value innodb_max_dirty_pages_pct_lwm=0
guarantees that a shutdown of an idle server will be fast. Users might
be surprised if normal shutdown suddenly became slower when upgrading
within a GA release series.

innodb_checkpoint_usec: Remove. The master task will no longer perform
periodic log checkpoints. It is the duty of the page cleaner thread.

log_sys.max_modified_age: Remove. The current span of the
buf_pool.flush_list expressed in LSN only matters for adaptive
flushing (outside the 'furious flushing' condition).
For the correctness of checkpoints, the only thing that matters is
the checkpoint age (log_sys.lsn - log_sys.last_checkpoint_lsn).
This run-time constant was also reported as log_max_modified_age_sync.

log_sys.max_checkpoint_age_async: Remove. This does not serve any
purpose, because the checkpoints will now be triggered by the page
cleaner thread. We will retain the log_sys.max_checkpoint_age limit
for engaging 'furious flushing'.

page_cleaner.slot: Remove. It turns out that
page_cleaner_slot.flush_list_time was duplicating
page_cleaner.slot.flush_time and page_cleaner.slot.flush_list_pass
was duplicating page_cleaner.flush_pass.
Likewise, there were some redundant monitor counters, because the
page cleaner thread no longer performs any buf_pool.LRU flushing, and
because there only is one buf_flush_page_cleaner thread.

buf_flush_sync_lsn: Protect writes by buf_pool.flush_list_mutex.

buf_pool_t::get_oldest_modification(): Add a parameter to specify the
return value when no persistent data pages are dirty. Require the
caller to hold buf_pool.flush_list_mutex.

log_buf_pool_get_oldest_modification(): Take the fall-back LSN
as a parameter. All callers will also invoke log_sys.get_lsn().

log_preflush_pool_modified_pages(): Replaced with buf_flush_wait_flushed().

buf_flush_wait_flushed(): Implement two limits. If not enough buffer pool
has been flushed, signal the page cleaner (unless innodb_flush_sync=OFF)
and wait for the page cleaner to complete. If the page cleaner
thread is not running (which can be the case durign shutdown),
initiate the flush and wait for it directly.

buf_flush_ahead(): If innodb_flush_sync=ON (the default),
submit a new buf_flush_sync_lsn target for the page cleaner
but do not wait for the flushing to finish.

log_get_capacity(), log_get_max_modified_age_async(): Remove, to make
it easier to see that af_get_pct_for_lsn() is not acquiring any mutexes.

page_cleaner_flush_pages_recommendation(): Protect all access to
buf_pool.flush_list with buf_pool.flush_list_mutex. Previously there
were some race conditions in the calculation.

buf_flush_sync_for_checkpoint(): New function to process
buf_flush_sync_lsn in the page cleaner thread. At the end of
each batch, we try to wake up any blocked buf_flush_wait_flushed().
If everything up to buf_flush_sync_lsn has been flushed, we will
reset buf_flush_sync_lsn=0. The page cleaner thread will keep
'furious flushing' until the limit is reached. Any threads that
are waiting in buf_flush_wait_flushed() will be able to resume
as soon as their own limit has been satisfied.

buf_flush_page_cleaner: Prioritize buf_flush_sync_lsn and do not
sleep as long as it is set. Do not update any page_cleaner statistics
for this special mode of operation. In the normal mode
(buf_flush_sync_lsn is not set for innodb_flush_sync=ON),
try to wake up once per second. No longer check whether
srv_inc_activity_count() has been called. After each batch,
try to perform a log checkpoint, because the best chances for
the checkpoint LSN to advance by the maximum amount are upon
completing a flushing batch.

log_t: Move buf_free, max_buf_free possibly to the same cache line
with log_sys.mutex.

log_margin_checkpoint_age(): Simplify the logic, and replace
a 0.1-second sleep with a call to buf_flush_wait_flushed() to
initiate flushing. Moved to the same compilation unit
with the only caller.

log_close(): Clean up the calculations. (Should be no functional
change.) Return whether flush-ahead is needed. Moved to the same
compilation unit with the only caller.

mtr_t::finish_write(): Return whether flush-ahead is needed.

mtr_t::commit(): Invoke buf_flush_ahead() when needed. Let us avoid
external calls in mtr_t::commit() and make the logic easier to follow
by having related code in a single compilation unit. Also, we will
invoke srv_stats.log_write_requests.inc() only once per
mini-transaction commit, while not holding mutexes.

log_checkpoint_margin(): Only care about log_sys.max_checkpoint_age.
Upon reaching log_sys.max_checkpoint_age where we must wait to prevent
the log from getting corrupted, let us wait for at most 1MiB of LSN
at a time, before rechecking the condition. This should allow writers
to proceed even if the redo log capacity has been reached and
'furious flushing' is in progress. We no longer care about
log_sys.max_modified_age_sync or log_sys.max_modified_age_async.
The log_sys.max_modified_age_sync could be a relic from the time when
there was a srv_master_thread that wrote dirty pages to data files.
Also, we no longer have any log_sys.max_checkpoint_age_async limit,
because log checkpoints will now be triggered by the page cleaner
thread upon completing buf_flush_lists().

log_set_capacity(): Simplify the calculations of the limit
(no functional change).

log_checkpoint_low(): Split from log_checkpoint(). Moved to the
same compilation unit with the caller.

log_make_checkpoint(): Only wait for everything to be flushed until
the current LSN.

create_log_file(): After checkpoint, invoke log_write_up_to()
to ensure that the FILE_CHECKPOINT record has been written.
This avoids ut_ad(!srv_log_file_created) in create_log_file_rename().

srv_start(): Do not call recv_recovery_from_checkpoint_start()
if the log has just been created. Set fil_system.space_id_reuse_warned
before dict_boot() has been executed, and clear it after recovery
has finished.

dict_boot(): Initialize fil_system.max_assigned_id.

srv_check_activity(): Remove. The activity count is counting transaction
commits and therefore mostly interesting for the purge of history.

BtrBulk::insert(): Do not explicitly wake up the page cleaner,
but do invoke srv_inc_activity_count(), because that counter is
still being used in buf_load_throttle_if_needed() for some
heuristics. (It might be cleaner to execute buf_load() in the
page cleaner thread!)

Reviewed by: Vladislav Vaintroub
2020-10-26 17:09:01 +02:00
Marko Mäkelä
bd67cb9284 MDEV-23399 fixup: Assertion bpage->in_file() failed
buf_flush_remove_pages(), buf_flush_dirty_pages(): Because
buf_page_t::state() is protected by buf_pool.mutex, which we
are not holding, the state may be BUF_BLOCK_REMOVE_HASH when
the page is being relocated. Let us relax these assertions
similar to buf_flush_validate_low().

The other in_file() assertions in buf0flu.cc look valid.
2020-10-26 17:09:01 +02:00
Eugene Kosov
31cde275c2 MDEV-23356 InnoDB: Failing assertion: field->col->mtype == type, crash or ASAN failures in row_sel_convert_mysql_key_to_innobase, InnoDB indexes are inconsistent after INDEX changes
innobase_rename_indexes_cache(): fix corruption of index cache. Index ids
help distinguish indexes when their names clash.

innobase_rename_indexes_cache(): fix corruption of index statistics table.
Use unique temporary names to avoid names clashing.

Reviewed by: Marko Mäkelä
2020-10-26 17:39:52 +03:00
Marko Mäkelä
59a0236da4 Cleanup: Speed up mariabackup --prepare
srv_start(): Avoid trx_lists_init_at_db_start() for normal
mariabackup --prepare without --export.
2020-10-26 15:06:32 +02:00
Marko Mäkelä
5999d5120e MDEV-23399 fixup: Avoid crash on Mariabackup shutdown
innodb_preshutdown(): Terminate the encryption threads before
the page cleaner thread can be shut down.

innodb_shutdown(): Always wait for the encryption threads and
page cleaner to shut down.

srv_shutdown_all_bg_threads(): Wait for the encryption threads and
the page cleaner to shut down. (After an aborted startup,
innodb_shutdown() would not be called.)

row_get_background_drop_list_len_low(): Remove.

os_thread_count: Remove. Alternatively, at the end of
srv_shutdown_all_bg_threads() we could try to wait longer
for the count to reach 0. On some platforms, an assertion
os_thread_count==0 could fail even after a small delay,
even though in the core dump all threads would have exited.

srv_shutdown_threads(): Renamed from srv_shutdown_all_bg_threads().
Do not wait for the page cleaner to shut down, because the later
innodb_shutdown(), which may invoke
logs_empty_and_mark_files_at_shutdown(), assumes that it exists.
2020-10-26 15:05:41 +02:00
Sachin Agarwal
1ff8588c3f Bug #31228694 FTS QUERY WITH LIMIT HIT AN ASSERT
Problem:
1. The server terminates abnormally when phrase search doesn't
filter out doc_ids correctly. This problem has been fixed in bug
2. Wrong query result: It's a regression from the bug #22709692 fix.
This fix optimize full-text search query with limit clause.
when FTS expression involves only union operation, we fetch only
number of doc_ids specified with the limit clause.
Fulltext phrase search is not an union operation and we consider
phrase search with plugin parser a union operation.
In phrase search with limit clause, we fetch limited doc_ids for
each token and if any of the selected doc_id does not contain all
tokens in correct order then we do not include that row_id in the
result set.
Therefore phrase search gets fewer number of rows than the qualified
rows exist in the table.

Fix:
Added a condition that phrase search with plugin parser is not a
union operation.

RB: 24925
Reviewed by : Annamalai Gurusami <annamalai.gurusami@oracle.com>

This is a cherry-pick of
mysql/mysql-server@5549920b7a
without a test case, because the test case depends on an n-gram
tokenizer that will be missing from MariaDB until MDEV-10267 is added.
2020-10-26 13:32:30 +02:00
Sachin Agarwal
e391417f0f Bug #30933728 INNODB FTS PHRASE SEARCH HIT AN ASSERT
Problem:
In Full-text phrase search, we filter out row that do not contain
all the tokens in the phrase.
If we do not filter out doc_id that doesn't appear in all the
token's doc_id lists then we hit an assert.

Fix:
if any of the token has last doc_id equal to ith doc_id of the first
token doc_id list then filter out rest of the higher doc_ids.

RB: 24909
Reviewed by : Annamalai Gurusami <annamalai.gurusami@oracle.com>

This is a cherry-pick of
mysql/mysql-server@5aa075277d
but without a test case, because the test case depends on an n-gram
tokenizer that will be missing from MariaDB until MDEV-10267 is added.
2020-10-26 12:21:29 +02:00
Oleksandr Byelkin
784473b986 Merge remote-tracking branch 'connect/10.2' into 10.2 2020-10-26 09:08:44 +01:00
Daniel Black
045671d473 MDEV-23539: aws key plugin - fails to build
Recent gcc/clang versions failed to compile the existing code.

Updating a later upstream SDK version was simple and required
only implementing a flush method. This was left blank as
there was no strong requirement to keep the error log
atomic or durable.

Reviewed-by: wlad@mariadb.com

The upstream SDK version added a flush method which was simple
to complete.
2020-10-26 14:59:42 +11:00
Sergei Golubchik
2fdc50367c remove disable_abort_on_error from precedence.test
was left over from testing
2020-10-25 19:38:43 +01:00
Thirunarayanan Balathandayuthapani
3ba8f619e4 MDEV-23370 innodb_fts.innodb_fts_misc failed in buildbot, server crashed in dict_table_autoinc_destroy
This issue is caused by MDEV-22456 ad6171b91c. Fix involves the backported version of 10.4 patch
MDEV-22778 5f2628d1ee and few parts of
MDEV-17441 (e9a5f288f2).

dict_table_t::stats_latch_created: Removed

dict_table_t::stats_latch: make value member and always lock it for
simplicity even for stats cloned table.

zip_pad_info_t::mutex_created: Removed

zip_pad_info_t::mutex: make member value instead of pointer

os0once.h: Removed

dict_table_remove_from_cache_low(): Ensure that fts_free() is always
called, even if dict_mem_table_free() is deferred until
btr_search_lazy_free().

InnoDB would always zip_pad_info_t::mutex and
dict_table_t::autoinc_mutex, even for tables are not in
ROW_FORMAT=COMPRESSED nor include any AUTO_INCREMENT column.
2020-10-25 15:53:17 +05:30
Marko Mäkelä
987df9b37a MDEV-23720 Change innodb_log_optimize_ddl=OFF by default
MariaDB 10.2.2 inherited from MySQL 5.7 a perceived optimization
of ALTER TABLE, which skips the writing of redo log records.
In MDEV-16809 we introduced a parameter that allows the redo log to
be written, so that Mariabackup would not be impacted, but we kept
the MySQL 5.7 behaviour enabled by default (innodb_log_optimize_ddl=ON).

As noted in MDEV-19747 (Deprecate and ignore innodb_log_optimize_ddl,
implemented in MariaDB 10.5.1), omitting the redo log writes can
actually reduce performance, because we will have to wait for the data
pages to be written out. When the redo log file is configured to be
large enough, it actually can be much faster to write the redo log and
avoid the extra page flushing.

When the redo log is omitted (innodb_log_optimize_ddl=ON), also
Mariabackup may have to perform a lot of extra work, to re-copy the
entire data file if it is possible that any log was omitted during
the backup.

Starting with MariaDB 10.5.1, the parameter innodb_log_optimize_ddl
is deprecated and ignored. We hereby deprecate (but will not ignore)
the parameter in earlier versions as well.
2020-10-25 11:48:34 +02:00
Anel Husakovic
4e987b1c6b MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role
Reviewed-by: serg@mariadb.com
2020-10-24 16:32:07 +02:00
Jan Lindström
44c958dd7b Fix test failure on wsrep/variables test case. 2020-10-24 14:57:57 +03:00
Vlad Lesin
f679d72679 MDEV-24017: Blackhole : Specified key was too long; max key length is 1000 bytes
The maximum innodb key length is 3500 what is hardcoded in
ha_innobase::max_supported_key_length()). The maximum number of innodb indexes
is configured with MAX_INDEXES macro (see also MAX_KEY definition).

The same is currently implemented for blackhole storage engine.

Cherry picked from percona-server 0d90d81c3c507a6b1476246a405504f6e4ef9d4d
Original lp bug 1733049

Reviewed-by: daniel@mariadb.org
2020-10-24 22:43:21 +11:00
Sergei Golubchik
d8515c8d35 S3 plugin fails to load depending on loaded storage engines
storage engines are generally initialized in some random order
(by iterating the hash of plugin names).

S3 fails if it's initialized before Aria.

But it looks that while S3 needs Aria, it does not need
Aria to be initialized before S3. S3 copies maria_hton and then
overwrites every single member of it, so it can handle Aria being
initialized after S3.
2020-10-24 11:15:51 +02:00
Sergei Golubchik
e764d11829 fix occasisonal test failures: SELECT without ORDER BY 2020-10-24 11:15:51 +02:00
Sergei Golubchik
606a281162 MDEV-22100 TokuDB compilation error
fix TokuDB to compile with the perfschema
2020-10-24 11:15:51 +02:00
Sergei Golubchik
73a2ae9f89 MDEV-23764 Slave crashes in ha_storage_engine_is_enabled upon rename of view
view_pseudo_hton cannot be dereferenced
2020-10-24 11:15:51 +02:00
Sergei Golubchik
3cd4d8ddba parser optimization
first try to resolve the function name as a native function,
and only then look within pluggable data types
2020-10-24 11:15:51 +02:00
Sergei Golubchik
d8fbd463a0 don't use deprecated <target>_LIB_DEPENDS. CMP0073 2020-10-24 11:15:51 +02:00
Marko Mäkelä
64fe9d6d9a Do not leak memory in the skipped MDEV-23768 unit test 2020-10-24 11:10:37 +03:00
Oleksandr Byelkin
06af03677c Merge remote-tracking branch 'connect/10.1' into 10.1 2020-10-24 10:08:07 +02:00
Alexey Botchkov
9df99151da MDEV-23437 Item_func_json_objectagg::print is not implemented.
Fix the Item_func_json_objectagg::fix_fields to save the orig_args.
2020-10-24 00:16:56 +04:00