btr_cur_need_opposite_intention(): Check also page_zip_available()
so that we will escalate to exclusive index latch when a non-leaf
page may have to be split further due to ROW_FORMAT=COMPRESSED page
overflow.
Tested by: Matthias Leich
The InnoDB buffer pool and locking were heavily refactored in
MariaDB Server 10.6. Among other things, dict_sys.mutex was removed,
and the contended lock_sys.mutex was replaced with a combination of
lock_sys.latch and distributed latches in hash tables. Also, a
default value was changed to innodb_flush_method=O_DIRECT to improve
performance in write-heavy workloads.
One thing where an adjustment was missing is around the parameters
innodb_max_purge_lag (number of committed transactions waiting to
be purged), and innodb_max_purge_lag_delay
(maximum number of microseconds to delay a DML operation).
purge_coordinator_state::do_purge(): Pass the history_size to trx_purge()
and reset srv_dml_needed_delay if the history is empty.
Keep executing the loop non-stop as long as srv_dml_needed_delay is set.
trx_purge_dml_delay(): Made part of trx_purge().
Set srv_dml_needed_delay=0 when nothing can be purged (!n_pages_handled).
row_mysql_delay_if_needed(): Mimic the logic of
innodb_max_purge_lag_wait_update().
Reviewed by: Thirunarayanan Balathandayuthapani
log_free_check(): Assert that the caller must not hold
exclusive lock_sys.latch. This was the case for calls from
ibuf_delete_for_discarded_space(). This caused a deadlock with
another thread that would be holding a latch on a dirty page
that would need to be written so that the checkpoint would advance
and log_free_check() could return. That other thread was waiting
for a shared lock_sys.latch.
fil_delete_tablespace(): Do not invoke ibuf_delete_for_discarded_space()
because in DDL operations, we will be holding exclusive lock_sys.latch.
trx_t::commit(std::vector<pfs_os_file_t>&), innodb_drop_database(),
row_purge_remove_clust_if_poss_low(), row_undo_ins_remove_clust_rec(),
row_discard_tablespace_for_mysql():
Invoke ibuf_delete_for_discarded_space() on the deleted tablespaces after
releasing all latches.
page_cleaner_flush_pages_recommendation(): If dirty_pct is
between innodb_max_dirty_pages_pct_lwm
and innodb_max_dirty_pages_pct,
scale the effort relative to how close we are to
innodb_max_dirty_pages_pct.
The previous formula was missing a multiplication by 100.
Tested by: Axel Schwenke
buf_pool_t::page_cleaner_wakeup(): If for_LRU=true, wake up the page
cleaner immediately, also when it is in a timed wait. This avoids an
unnecessary delay of up to 1 second.
When commit a5a2ef079c
implemented asynchronous doublewrite, the writes via
the doublewrite buffer started to be counted incorrectly,
without multiplying them by innodb_page_size.
srv_export_innodb_status(): Correctly count the
Innodb_data_written.
buf_dblwr_t: Remove submitted(), because it is close to written()
and only Innodb_data_written was interested in it. According to
its name, it should count completed and not submitted writes.
Tested by: Axel Schwenke
os_aio_wait_until_no_pending_reads(), os_aio_wait_until_pending_writes():
Add a Boolean parameter to indicate whether the wait should be declared
in the thread pool.
buf_flush_wait(): The callers have already declared a wait, so let us
avoid doing that again, just call os_aio_wait_until_pending_writes(false).
buf_flush_wait_flushed(): Do not declare a wait in the rare case that
the buf_flush_page_cleaner thread has been shut down already.
buf_flush_page_cleaner(), buf_flush_buffer_pool(): In the code that runs
during shutdown, do not declare waits.
buf_flush_buffer_pool(): Remove a debug assertion that might fail.
What really matters here is buf_pool.flush_list.count==0.
buf_read_recv_pages(), srv_prepare_to_delete_redo_log_file():
Do not declare waits during InnoDB startup.
- This patch does the following:
git revert --no-commit 673243c893
git revert --no-commit 6c669b9586
git revert --no-commit bacaf2d4f4
git checkout HEAD mysql-test
git revert --no-commit 1fd7d3a9ad
Above command reverts MDEV-29277, MDEV-25581, MDEV-29342.
When binlog is enabled, trasaction takes a lot of time to do
sync operation on innodb fts table. This leads to block
of other transaction commit. To avoid this failure, remove
the fulltext sync operation during transaction commit. So
reverted MDEV-25581 related patches.
We filed MDEV-31105 to avoid the memory consumption
problem during fulltext sync operation.
- `mariadb-backup --backup` was fixed to fetch the value of the
@@aria_log_dir_path server variable and copy aria_log* files
from @@aria_log_dir_path directory to the backup directory.
Absolute and relative (to --datadir) paths are supported.
Before this change aria_log* files were copied to the backup
only if they were in the default location in @@datadir.
- `mariadb-backup --copy-back` now understands a new my.cnf and command line
parameter --aria-log-dir-path.
`mariadb-backup --copy-back` in the main loop in copy_back()
(when copying back from the backup directory to --datadir)
was fixed to ignore all aria_log* files.
A new function copy_back_aria_logs() was added.
It consists of a separate loop copying back aria_log* files from
the backup directory to the directory specified in --aria-log-dir-path.
Absolute and relative (to --datadir) paths are supported.
If --aria-log-dir-path is not specified,
aria_log* files are copied to --datadir by default.
- The function is_absolute_path() was fixed to understand MTR style
paths on Windows with forward slashes, e.g.
--aria-log-dir-path=D:/Buildbot/amd64-windows/build/mysql-test/var/...
The motivation of this change is to allow undo pages for temporary tables
to be marked free as often as possible, so that we can avoid buf_pool.LRU
eviction (and writes) of undo pages that contain data that is
no longer needed. For temporary tables, no MVCC or purge of history
is needed, and reusing cached undo log pages might not help that much.
It is possible that this may cause some performance regression due to
more frequent allocation and freeing of undo log pages, but I only
measured a performance improvement.
trx_write_serialisation_history(): Never cache temporary undo log pages.
trx_undo_reuse_cached(): Assert that the rollback segment is persistent.
trx_undo_assign_low(): Add template<bool is_temp>. Never invoke
trx_undo_reuse_cached() for temporary tables.
Tested by: Matthias Leich
Let us remove explicit updates of MONITOR_NUM_UNDO_SLOT_USED
and MONITOR_NUM_UNDO_SLOT_CACHED, and let us compute the rough values
from trx_sys.rseg_array[] on demand.
trx_purge_truncate_rseg_history(): If all other conditions for
invoking trx_purge_remove_log_hdr() hold, but the state is
TRX_UNDO_CACHED instead of TRX_UNDO_TO_PURGE, detach and free it.
Tested by: Matthias Leich
buf_LRU_get_free_block(): Always wake up the page cleaner if needed
before exiting the inner loop.
srv_prepare_to_delete_redo_log_file():
Replace a debug assertion with a wait in debug builds.
Starting with commit 7e31a8e7fa
the debug assertion ut_ad(!os_aio_pending_writes())
could occasionally fail, while it would hold in core dumps of crashes.
The failure can be reproduced more easily by adding a sleep to the
write completion callback function, right before releasing to
write_slots.
srv_start(): Remove a bogus debug assertion
ut_ad(!os_aio_pending_writes()) that could fail in
mariadb-backup --prepare. In an rr replay trace, we had
buf_pool.flush_list.count==0 but write_slots->m_cache.m_pos==1
and buf_page_t::write_complete() was executing u_unlock().
buf_flush_wait_flushed(): Correct the logic for registering a wait
around buf_flush_wait() that
commit a091d6ac4e
recently broke. This should be easily repeatable when using a
non-default startup parameter:
thread-handling=pool-of-threads
trx_purge_free_segment(): The buffer-fix only prevents a block from
being freed completely from the buffer pool, but it will not prevent
the block from being evicted. Recheck the page identifier after
acquiring an exclusive page latch. If it has changed, backtrack and
invoke buf_page_get_gen() to look up the page normally.
io_callback(): Process the request before releasing the write slot.
Before commit a091d6ac4e
when we had a duplicated counter for writes, either ordering was fine.
Now, correctness depends on os_aio_wait_until_no_pending_writes().
buf_flush_LRU_list_batch(): When evicting clean pages,
release and reacquire the buf_pool.mutex after every 32 pages.
Also, eliminate some conditional branches.
fil_space_t::create(), fil_space_t::add(): Expect the caller to
acquire and release fil_system.mutex. In this way, creating a tablespace
and adding the first (usually only) data file will be atomic.
recv_sys_t::recover_deferred(): Correctly protect some changes by
holding fil_system.mutex.
Tested by: Matthias Leich
trx_assign_rseg_low(): Let us restore the debug variable look_for_rollover
to avoid assertion failures when a server that was created with
multiple undo tablespaces is being started with innodb_undo_tablespaces=0.
stored externally
row_merge_buf_add(): Has strict assert that fixed length mismatch
shouldn't happen while rebuilding the redundant row format table
btr_index_rec_validate(): Fixed size column can be stored externally.
So sum of inline stored length and external stored length of the
column should be equal to total column length
This issue happens when race condition happens when DDL
and fts optimize thread. DDL adds the new index to fts cache.
At the same time, fts optimize thread clears the cache
and reinitialize it. Take cache init lock before reinitializing
the cache. fts_sync_commit() should take dict_sys mutex
to avoid the deadlock with create index.
trx_assign_rseg_low(): Simplify the debug check.
trx_rseg_t::reinit(): Reset the skip_allocation() flag.
This logic was broken in the merge
commit 3e2ad0e918
of commit 0de3be8cfd
(that is, innodb_undo_log_truncate=ON would never be "completed").
Tested by: Matthias Leich
fil_delete_tablespace() stores file handle in local variable and calls
mtr_t::commit_file()=>fil_system_t::detach(..., detach_handle=true), which
sets space->chain.start->handle = OS_FILE_CLOSED. fil_system_t::detach()
is invoked under fil_system.mutex.
But before the mutex is acquired some parallel thread can change
space->chain.start->handle. fil_delete_tablespace() returns value, stored
in local variable, i.e. wrong value.
File handle can be closed, for example, from buf_flush_space() when the
limit of innodb_open_files exceded and fil_space_t::get() causes
fil_space_t::try_to_close() call.
fil_space_t::try_to_close() is executed under fil_system.mutex. And
mtr_t::commit_file() locks it for fil_system_t::detach() call.
fil_system_t::detach() returns detached file handle if its argument
detach_handle is true. The fix is to let mtr_t::commit_file() to pass
that detached file handle to fil_delete_tablespace().
Post-push fix.
10.5 MDEV-30775 fix inserts just opened tablespace just after the element
which fil_system.space_list_last_opened points to.
In MDEV-25223 fil_system_t::space_list was changed from UT_LIST to
ilist. ilist<...>::insert(iterator pos, reference value) inserts element
to list before pos.
But it was not taken into account during 10.5->10.6 merge in
85cbfaefee, and the fix
does not work properly, i.e. it inserted just opened tablespace to the
position preceding fil_system.space_list_last_opened.
tpool::cache::m_mtx: Add PERFORMANCE_SCHEMA instrumentation
(wait/synch/mutex/innodb/tpool_cache_mutex). This covers the
InnoDB read_slots and write_slots for asynchronous data page I/O.
- This issue caused by race condition between drop thread
and fil_encrypt_thread. fil_encrypt_thread closes
the tablespace if the number of opened files
exceeds innodb_open_files. fil_node_open_file()
closes the tablespace which are open and it doesn't
have pending operations. At that time, InnoDB drop tries
to write the redo log for the file delete operation.
It throws the bad file descriptor error.
- When trying to close the file, InnoDB should check
whether the table is going to be dropped.
- Adding a new argument "flag" to MY_COLLATION_HANDLER::strnncollsp_nchars()
and a flag MY_STRNNCOLLSP_NCHARS_EMULATE_TRIMMED_TRAILING_SPACES.
The flag defines if strnncollsp_nchars() should emulate trailing spaces
which were possibly trimmed earlier (e.g. in InnoDB CHAR compression).
This is important for NOPAD collations.
For example, with this input:
- str1= 'a ' (Latin letter a followed by one space)
- str2= 'a ' (Latin letter a followed by two spaces)
- nchars= 3
if the flag is given, strnncollsp_nchars() will virtually restore
one trailing space to str1 up to nchars (3) characters and compare two
strings as equal:
- str1= 'a ' (one extra trailing space emulated)
- str2= 'a ' (as is)
If the flag is not given, strnncollsp_nchars() does not add trailing
virtual spaces, so in case of a NOPAD collation, str1 will be compared
as less than str2 because it is shorter.
- Field_string::cmp_prefix() now passes the new flag.
Field_varstring::cmp_prefix() and Field_blob::cmp_prefix() do
not pass the new flag.
- The branch in cmp_whole_field() in storage/innobase/rem/rem0cmp.cc
(which handles the CHAR data type) now also passed the new flag.
- Fixing UCA collations to respect the new flag.
Other collations are possibly also affected, however
I had no success in making an SQL script demonstrating the problem.
Other collations will be extended to respect this flags in a separate
patch later.
- Changing the meaning of the last parameter of Field::cmp_prefix()
from "number of bytes" (internal length)
to "number of characters" (user visible length).
The code calling cmp_prefix() from handler.cc was wrong.
After this change, the call in handler.cc became correct.
The code calling cmp_prefix() from key_rec_cmp() in key.cc
was adjusted according to this change.
- Old strnncollsp_nchar() related tests in unittest/strings/strings-t.c
now pass the new flag.
A few new tests also were added, without the flag.
Sequence objects are implemented using special tables.
These tables do not have primary key and only one row.
NEXTVAL is basically update from existing value to new
value. In Galera this could mean that two write-sets
from different nodes do not conflict and this could
lead situation where write-sets are executed
concurrently and possibly in wrong seqno order.
This is fixed by using table-level exclusive key for
SEQUENCE updates. Note that this naturally works
correctly only if InnoDB storage engine is used for
sequence.
This fix does not contain a test case because while
it is possible to syncronize appliers using dbug_sync
it was too hard to syncronize MDL-lock requests to
exact objects. Testing done for this fix is documented
on MDEV.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
Let us make innodb_buffer_pool_filename a read-only variable
so that a malicious user cannot cause an important file to be
deleted on InnoDB shutdown. An attempt to delete a directory
will fail because it is not a regular file, but what if the
variable pointed to (say) ibdata1, ib_logfile0 or some *.ibd file?
It does not seem to make much sense for this parameter to be
configurable in the first place, but we will not change that in order
to avoid breaking compatibility.
handle_slave_io(), handle_slave_sql(), os_thread_exit():
Remove a redundant pthread_exit(nullptr) call, because it
would cause SIGSEGV.
mysql_print_status(): Add MEM_MAKE_DEFINED() to work around
some missing instrumentation around mallinfo2().
que_graph_free_stat_list(): Invoke que_node_get_next(node) before
que_graph_free_recursive(node). That is the logical and
MSAN_OPTIONS=poison_in_dtor=1 compatible way of freeing memory.
ins_node_t::~ins_node_t(): Invoke mem_heap_free(entry_sys_heap).
que_graph_free_recursive(): Rely on ins_node_t::~ins_node_t().
fts_t::~fts_t(): Invoke mem_heap_free(fts_heap).
fts_free(): Replace with direct calls to fts_t::~fts_t().
The failures in free_root() due to MSAN_OPTIONS=poison_in_dtor=1
will be covered in MDEV-30942.
The solution is to suppress error messages for missing tablespaces if
mariabackup is launched with "--prepare --export" options.
"mariabackup --prepare --export" invokes itself with --mysqld parameter.
If the parameter is set, then it starts server to feed "FLUSH TABLES ...
FOR EXPORT;" queries for exported tablespaces. This is "normal" server
start, that's why new srv_operation value is introduced.
Reviewed by Marko Makela.
Problem:
========
- InnoDB replace statement returns can't find record as result during
bulk insert operation. InnoDB returns DB_END_OF_INDEX blindly when
bulk transaction is visible to current transaction even though
the search tuple is inserted as a part of current replace statement.
Solution:
=========
row_search_mvcc(): InnoDB should allow the transaction to read
all the rows when innodb intends to do any locking on the
record even though bulk insert transaction changes are
visible to the current transaction
buf_dblwr_t::init(), buf_dblwr_t::close(): Cover also write_cond,
which was added in commit a55b951e60
without explicit initialization. On GNU/Linux, PTHREAD_COND_INITIALIZER
is a zero-initializer. That is why the default zero initialization
happened to work on that platform.
btr_cur_upd_rec_in_place(): Avoid calling page_zip_write_rec() if we
are not modifying any fields that are stored in compressed format.
btr_cur_update_in_place_zip_check(): New function to check if a
ROW_FORMAT=COMPRESSED record can actually be updated in place.
btr_cur_pessimistic_update(): If the BTR_KEEP_POS_FLAG is not set
(we are in a ROLLBACK and cannot write any BLOBs), ignore the potential
overflow and let page_zip_reorganize() or page_zip_compress() handle it.
This avoids a failure when an attempted UPDATE of an NULL column to 0 is
rolled back. During the ROLLBACK, we would try to move a non-updated
long column to off-page storage in order to avoid a compression failure
of the ROW_FORMAT=COMPRESSED page.
page_zip_write_trx_id_and_roll_ptr(): Remove an assertion that would fail
in row_upd_rec_in_place() because the uncompressed page would already
have been modified there.
This is a 10.5 version of commit ff3d4395d8
(different because of commit 08ba388713).
