Problem was that if tablespace was encrypted we try to copy
also page 0 from read buffer to write buffer that are in
that case the same memory area.
fil_iterate
When tablespace is encrypted or compressed its
first page (i.e. page 0) is not encrypted or
compressed and there is no need to copy buffer.
If innodb_fast_shutdown<2, all transactions of active connections
will be rolled back on shutdown. This can take a long time, and
the systemd shutdown timeout should be extended during the wait.
logs_empty_and_mark_files_at_shutdown(): Extend the timeout when
waiting for other threads to complete.
This reverts commit 76ec37f522.
This behaviour change will be done separately in:
MDEV-15832 With innodb_fast_shutdown=3, skip the rollback
of connected transactions
fil_crypt_read_crypt_data(): Do not attempt to read the tablespace
if the file is unaccessible due to a pending DDL operation, such as
renaming the file or DROP TABLE or TRUNCATE TABLE. This is only
reducing the probability of the race condition, not completely
preventing it.
Problem was that key rotation from encrypted to unecrypted was skipped
when encryption is disabled (i.e. set global innodb-encrypt-tables=OFF).
fil_crypt_needs_rotation
If encryption is disabled (i.e. innodb-encrypt-tables=off)
and there is tablespaces using default encryption (e.g.
system tablespace) that are still encrypted state we need
to rotate them from encrypted state to unencrypted state.
buf_flush_remove(): Disable the output for now, because we
certainly do not want this after every page flush on shutdown.
It must be rate-limited somehow. There already is a timeout
extension for waiting the page cleaner to exit in
logs_empty_and_mark_files_at_shutdown().
log_write_up_to(): Use correct format.
srv_purge_should_exit(): Move the timeout extension to the
appropriate place, from one of the callers.
Use systemd EXTEND_TIMEOUT_USEC to advise systemd of progress
Move towards progress measures rather than pure time based measures.
Progress reporting at numberious shutdown/startup locations incuding:
* For innodb_fast_shutdown=0 trx_roll_must_shutdown() for rolling back incomplete transactions.
* For merging the change buffer (in srv_shutdown(bool ibuf_merge))
* For purging history, srv_do_purge
Thanks Marko for feedback and suggestions.
Suggested by Marko on github pr #576
buf_all_freed only needs to be called once, not 3 times.
buf_all_freed will always return TRUE if it returns.
It will crash if any page was not flushed so its effectively
an assert anyway.
The following calls are likely redundant and could be removed:
fil_flush_file_spaces(FIL_TYPE_TABLESPACE);
fil_flush_file_spaces(FIL_TYPE_LOG);
row_undo_step(): If fast shutdown has been requested, abort the
rollback of any non-DDL transactions. Starting with MDEV-12323,
we aborted the rollback of recovered transactions. These
transactions would be rolled back on subsequent server startup.
trx_roll_report_progress(): Renamed from trx_roll_must_shutdown(),
now that the shutdown check has been moved to the only caller.
fil_iterate(), fil_tablespace_iterate(): Replace os_file_read()
with os_file_read_no_error_handling().
os_file_read_func(), os_file_read_no_error_handling_func():
Do not retry partial reads. There used to be an infinite amount
of retries. Because InnoDB extends both data and log files upfront,
partial reads should be impossible during normal operation.
Initialize block.page.zip only once.
PageConverter::update(): Initialize m_page_zip_ptr
as late as possible.
(We should really remove it at some point.)
PageConverter::operator(): Refer to block->page.zip instead of
m_page_zip_ptr.
AbstractCallback::get_frame(): Define static. Refer
to block->page.zip.data directly.
fil_iterate(): Refer to block->page.zip.data directly.
fil_tablespace_iterate(): Initialize block.page.zip.data as soon
as possible.
Reduce unnecessary inter-module calls for IMPORT TABLESPACE.
Move some IMPORT-related code from fil0fil.cc to row0import.cc.
PageCallback: Remove. Make AbstractCallback the base class.
PageConverter: Define some member functions inline.
assert on UTF-8 columns
Problem:
=======
(1) Multi-byte character cases are not considered during prefix index
cluster optimization check. It leads to fetch of improper results during
read operation.
(2) Strict assert in row_sel_field_store_in_mysql_format_func and it asserts
for prefix index record to mysql conversion.
Solution:
========
(1) Consider the case of multi-byte character during prefix index
cluster optimization check.
(2) Relax the assert in row_sel_field_store_in_mysql_format_func to allow
prefix index record to mysql format conversion.
The patch is taken from
1eee538087
Learn both valgrind and asan to catch this bug:
mem_heap_t* heap = mem_heap_create(1024);
byte* p = reinterpret_cast<byte*>(heap) + sizeof(mem_heap_t);
*p = 123;
Overflows of the last allocation in a block will be catched too.
mem_heap_create_block(): poison newly allocated memory
Unlike commit a54abf0175 claimed,
the caller of THD::awake() may actually hold the InnoDB lock_sys->mutex.
That commit introduced a deadlock of threads in the replication slave
when running the test rpl.rpl_parallel_optimistic_nobinlog.
lock_trx_handle_wait(): Expect the callers to acquire and release
lock_sys->mutex and trx->mutex.
innobase_kill_query(): Restore the logic for conditionally acquiring
and releasing the mutexes. THD::awake() can be called from inside
InnoDB while holding one or both mutexes, via thd_report_wait_for() and
via wsrep_innobase_kill_one_trx().
ha_innobase::unlock_row(): Use a relaxed version of the
trx_state_eq() debug assertion, because rr_unlock_row()
may be invoked after an error has been already reported
and the transaction has been rolled back.
By definition, c_lock->trx->lock.wait_lock==c_lock cannot hold.
That is, the owner transaction of a lock cannot be waiting for that
particular lock. It must have been waiting for some other lock.
Remove the dead code related to that. Also, test c_lock for NULLness
only once.
As this is the only moderately critical fopened for writing file,
create an alternate path to use open and fdopen for non-glibc platforms
that support O_CLOEXEC (BSDs).
Tested on Linux (by modifing the GLIBC defination) to take this
alternate path:
$ cd /proc/23874
$ more fdinfo/71
pos: 0
flags: 02100001
mnt_id: 24
$ ls -la fd/71
l-wx------. 1 dan dan 64 Mar 14 13:30 fd/71 -> /dev/shm/var_auto_i7rl/mysqld.1/data/ib_buffer_pool.incomplete
fts_sync(): If the dict_table_t::to_be_dropped flag is set,
do not "goto begin_sync".
Also, clean up the way how dict_index_t::index_fts_syncing
is cleared.
It looks like this regression was introduced by merging
Oracle Bug #24938374 MYSQL CRASHED AFTER LONG WAIT ON DICT OPERATION LOCK
WHILE SYNCING FTS INDEX
068f8261d4
from MySQL 5.6.38 into MariaDB 10.0.33, 10.1.29, 10.2.10.
The same hang is present in MySQL 5.7.20.
InnoDB in Debian uses utf8mb4 as default character set since
version 10.0.20-2. This leads to major pain due to keys longer
than 767 bytes.
MariaDB 10.2 (and MySQL 5.7) introduced the setting
innodb_default_row_format that is DYNAMIC by default. These
versions also changed the default values of the parameters
innodb_large_prefix=ON and innodb_file_format=Barracuda.
This would allow longer column index prefixes to be created.
The original purpose of these parameters was to allow InnoDB
to be downgraded to MySQL 5.1, which is long out of support.
Every InnoDB version since MySQL 5.5 does support operation
with the relaxed limits.
We backport the parameter innodb_default_row_format to
MariaDB 10.1, but we will keep its default value at COMPACT.
This allows MariaDB 10.1 to be configured so that CREATE TABLE
is less likely to encounter a problem with the limitation:
loose_innodb_large_prefix=ON
loose_innodb_default_row_format=DYNAMIC
(Note that the setting innodb_large_prefix was deprecated in
MariaDB 10.2 and removed in MariaDB 10.3.)
The only observable difference in the behaviour with the default
settings should be that ROW_FORMAT=DYNAMIC tables can be created
both in the system tablespace and in .ibd files, no matter what
innodb_file_format has been assigned to. Unlike MariaDB 10.2,
we are not changing the default value of innodb_file_format,
so ROW_FORMAT=COMPRESSED tables cannot be created without
changing the parameter.
This performance regression was introduced in the MariaDB 10.1
file format incompatibility bug fix MDEV-11623 (MariaDB 10.1.21
and MariaDB 10.2.4) and partially fixed in MariaDB 10.1.25 in
MDEV-12610 without adding a regression test case.
On a normal startup (without crash recovery), InnoDB should not read
every .ibd data file, because this is slow. Like in MySQL, for now,
InnoDB will still open every data file (without reading), and it
will read every .ibd file for which an .isl file exists, or the
DATA DIRECTORY attribute has been specified for the table.
The test case shuts down InnoDB, moves data files, replaces them
with garbage, and then restarts InnoDB, expecting no messages to
be issued for the garbage files. (Some messages will for now be
issued for the table that uses the DATA DIRECTORY attribute.)
Finally, the test shuts down the server, restores the old data files,
and restarts again to drop the tables.
fil_open_single_table_tablespace(): Remove the condition on flags,
and only call fsp_flags_try_adjust() if validate==true
(reading the first page has been requested). The only caller with
validate==false is at server startup when we are processing all
records from SYS_TABLES. The flags passed to this function are
actually derived from SYS_TABLES.TYPE and SYS_TABLES.N_COLS,
and there never was any problem with SYS_TABLES in MariaDB 10.1.
The problem that MDEV-11623 was that incorrect tablespace flags
were computed and written to FSP_SPACE_FLAGS.
lock_trx_release_locks(): Relax a debug assertion to allow
recovered TRX_STATE_COMMITTED_IN_MEMORY transactions.
trx_commit_in_memory(): Add DEBUG_SYNC instrumentation.
trx_undo_insert_cleanup(): Skip persistent changes if innodb_read_only
is set. This should only happen when a recovered committed transaction
would be cleaned up at shutdown.
Before that line there is call to buf_page_get_gen that could
return block = NULL when decrypting a page fails. However,
we should set error to be != DB_SUCCESS also. In error log
there was error about decompression but in that code there
is one case where error is not set correctly.
PageConverter::adjust_cluster_record(): Instead of writing
the invalid value DB_ROLL_PTR=0, write a value that indicates
a fresh insert, that is, prevents the DB_ROLL_PTR from being
dereferenced in any circumstances.
It can be argued that IMPORT TABLESPACE should actually
update the dict_index_t::trx_id to prevent older transactions
from accessing the table, similar to what I did on table
rebuild in MySQL 5.6.6 in
03f81a55f2
Previously, the function could theoretically return an uninitialized
value if the system tablespace contained no data files. It should be
impossible for InnoDB to start up in such scenario.
The merge omitted some InnoDB and XtraDB conflict resolutions,
most notably, failing to merge the fix of MDEV-12173.
ibuf_merge_or_delete_for_page(), lock_rec_block_validate():
Invoke fil_space_acquire_silent() instead of fil_space_acquire().
This fixes MDEV-12173.
wsrep_debug, wsrep_trx_is_aborting(): Removed unused declarations.
_fil_io(): Remove. Instead, declare default parameters for the XtraDB
fil_io().
buf_read_page_low(): Declare default parameters, and clean up some
callers.
os_aio(): Correct the macro that is defined when !UNIV_PFS_IO.
