InnoDB crash recovery used to read every data page for which
redo log exists. This is unnecessary for those pages that are
initialized by the redo log. If a newly created page is corrupted,
recovery could unnecessarily fail. It would suffice to reinitialize
the page based on the redo log records.
To add insult to injury, InnoDB crash recovery could hang if it
encountered a corrupted page. We will fix also that problem.
InnoDB would normally refuse to start up if it encounters a
corrupted page on recovery, but that can be overridden by
setting innodb_force_recovery=1.
Data pages are completely initialized by the records
MLOG_INIT_FILE_PAGE2 and MLOG_ZIP_PAGE_COMPRESS.
MariaDB 10.4 additionally recognizes MLOG_INIT_FREE_PAGE,
which notifies that a page has been freed and its contents
can be discarded (filled with zeroes).
The record MLOG_INDEX_LOAD notifies that redo logging has
been re-enabled after being disabled. We can avoid loading
the page if all buffered redo log records predate the
MLOG_INDEX_LOAD record.
For the internal tables of FULLTEXT INDEX, no MLOG_INDEX_LOAD
records were written before commit aa3f7a107c.
Hence, we will skip these optimizations for tables whose
name starts with FTS_.
This is joint work with Thirunarayanan Balathandayuthapani.
fil_space_t::enable_lsn, file_name_t::enable_lsn: The LSN of the
latest recovered MLOG_INDEX_LOAD record for a tablespace.
mlog_init: Page initialization operations discovered during
redo log scanning. FIXME: This really belongs in recv_sys->addr_hash,
and should be removed in MDEV-19176.
recv_addr_state: Add the new state RECV_WILL_NOT_READ to
indicate that according to mlog_init, the page will be
initialized based on redo log record contents.
recv_add_to_hash_table(): Set the RECV_WILL_NOT_READ state
if appropriate. For now, we do not treat MLOG_ZIP_PAGE_COMPRESS
as page initialization. This works around bugs in the crash
recovery of ROW_FORMAT=COMPRESSED tables.
recv_mark_log_index_load(): Process a MLOG_INDEX_LOAD record
by resetting the state to RECV_NOT_PROCESSED and by updating
the fil_name_t::enable_lsn.
recv_init_crash_recovery_spaces(): Copy fil_name_t::enable_lsn
to fil_space_t::enable_lsn.
recv_recover_page(): Add the parameter init_lsn, to ignore
any log records that precede the page initialization.
Add DBUG output about skipped operations.
buf_page_create(): Initialize FIL_PAGE_LSN, so that
recv_recover_page() will not wrongly skip applying
the page-initialization record due to the field containing
some newer LSN as a leftover from a different page.
Do not invoke ibuf_merge_or_delete_for_page() during
crash recovery.
recv_apply_hashed_log_recs(): Remove some unnecessary lookups.
Note if a corrupted page was found during recovery.
After invoking buf_page_create(), do invoke
ibuf_merge_or_delete_for_page() via mlog_init.ibuf_merge()
in the last recovery batch.
ibuf_merge_or_delete_for_page(): Relax a debug assertion.
innobase_start_or_create_for_mysql(): Abort startup if
a corrupted page was found during recovery. Corrupted pages
will not be flagged if innodb_force_recovery is set.
However, the recv_sys->found_corrupt_fs flag can be set
regardless of innodb_force_recovery if file names are found
to be incorrect (for example, multiple files with the same
tablespace ID).
Similar to what was done in commit aa3f7a107c
for FULLTEXT INDEX, we must ensure that MLOG_INDEX_LOAD records will always
be written if redo logging was disabled.
row_merge_build_indexes(): Invoke row_merge_write_redo() also when
online operation is not being executed or an error occurs.
In case of an error, invoke flush_observer->interrupted() so that
the pages will not be flushed but merely evicted from the buffer pool.
Before resuming redo logging, it is crucial for the correctness of
mariabackup and InnoDB crash recovery to flush or evict all affected pages
and to write MLOG_INDEX_LOAD records.
innobase_init(): Add a missing space to a warning message.
Apparently, this message was corrupted in MariaDB 10.2.2 in
commit fec844aca8 related to a
conflict resolution when applying a change from MySQL 5.7.12.
The regression that was reported in MDEV-19212 occurred due to use
of macros that did not ensure that the arguments have compatible
types.
ut_2pow_remainder(), ut_2pow_round(), ut_calc_align(): Define as
inline function templates.
UT_CALC_ALIGN(): Define as a macro, because this is used in
compile_time_assert(). Only starting with C++11 (MariaDB 10.4)
we could define the inline functions as constexpr.
os_mem_alloc_large(): Invoke the macro ut_2pow_round() with the
correct argument type.
innobase_large_page_size, innobase_use_large_pages,
os_use_large_pages, os_large_page_size: Remove.
Simply refer to opt_large_page_size, my_use_large_pages.
btr_page_free(): Renamed from btr_page_free_low().
If scrubbing is enabled, zero out the page with proper redo logging.
Only pass ahi=true to fseg_free_page() if the page is actually indexed.
fil_space_t::modify_check(): Renamed from fsp_space_modify_check().
fsp_init_file_page(): Define inline.
This is a follow-up to MDEV-18733. As part of that fix, we made
dict_check_sys_tables() skip tables that would be dropped by
row_mysql_drop_garbage_tables().
DICT_ERR_IGNORE_DROP: A new mode where the file should not be attempted
to be opened.
dict_load_tablespace(): Do not try to load the tablespace if
DICT_ERR_IGNORE_DROP has been specified.
row_mysql_drop_garbage_tables(): Pass the DICT_ERR_IGNORE_DROP mode.
fil_space_for_table_exists_in_mem(): Remove a parameter.
The only caller that passed print_error_if_does_not_exist=true
was row_drop_single_table_tablespace().
The recv_sys data structures are accessed not only from the thread
that executes InnoDB plugin initialization, but also from the
InnoDB I/O threads, which can invoke recv_recover_page().
Assert that sufficient concurrency control is in place.
Some code was accessing recv_sys data structures without
holding recv_sys->mutex.
recv_recover_page(bpage): Refactor the call from buf_page_io_complete()
into a separate function that performs necessary steps. The
main thread was unnecessarily releasing and reacquiring recv_sys->mutex.
recv_recover_page(block,mtr,recv_addr): Pass more parameters from
the caller. Avoid redundant lookups and computations. Eliminate some
redundant variables.
recv_get_fil_addr_struct(): Assert that recv_sys->mutex is being held.
That was not always the case!
recv_scan_log_recs(): Acquire recv_sys->mutex for the whole duration
of the function. (While we are scanning and buffering redo log records,
no pages can be read in.)
recv_read_in_area(): Properly protect access with recv_sys->mutex.
recv_apply_hashed_log_recs(): Check recv_addr->state only once,
and continuously hold recv_sys->mutex. The mutex will be released
and reacquired inside recv_recover_page() and recv_read_in_area(),
allowing concurrent processing by buf_page_io_complete() in I/O threads.
The record MLOG_INDEX_LOAD is supposed to be written to indicate that
some page modifications bypassed redo logging, and that redo logging
is now re-enabled. It was not written for fulltext indexes during
ALTER TABLE.
row_merge_write_redo(): Declare globally. Assert that the index
is neither a spatial nor fulltext index.
recv_mlog_index_load(): Observe a MLOG_INDEX_LOAD operation.
recv_parse_log_recs(): Handle MLOG_INDEX_LOAD also in multi-record
mini-transactions. Because of this omission, we should keep writing
MLOG_INDEX_LOAD in single-record mini-transactions, because older
versions of Mariabackup would fail.
row_fts_merge_insert(): Write MLOG_INDEX_LOAD for the auxiliary
tables of fulltext indexes.
The record MLOG_ZIP_PAGE_COMPRESS is similar to MLOG_INIT_FILE_PAGE2
that it contains all the information needed to initialize the page.
Like for the other record, do initialize the entire page on recovery.
The page_size argument to buf_page_get_gen() only matters when the
page is going to be loaded into the buffer pool. Allow callers to
pass a dummy parameter when using BUF_GET_IF_IN_POOL (which would
return NULL if the block is not in the buffer pool).
btr_root_get(): Ignore the root->page.encrypted flag.
The purpose of this flag is questionable since
commit 8c43f96388.
btr_validate_index(): Avoid crash if btr_root_get() returns NULL.
Normally, InnoDB is not in the process of executing crash recovery.
Provide a hint to the compiler that the recovery-related code paths
are rarely executed.
If InnoDB crash recovery was needed, the InnoDB function srv_start()
would invoke extra validation, reading something from every InnoDB
data file. This should be unnecessary now that MDEV-14717 made
RENAME operations crash-safe inside InnoDB (which can be
disabled in MariaDB 10.2 by setting innodb_safe_truncate=OFF).
dict_check_sys_tables(): Skip tables that would be dropped by
row_mysql_drop_garbage_tables(). Perform extra validation only
if innodb_safe_truncate=OFF, innodb_force_recovery=0 and
crash recovery was needed.
dict_load_table_one(): Validate the root page of the table.
In this way, we can deny access to corrupted or mismatching tables
not only after crash recovery, but also after a clean shutdown.
In MariaDB, InnoDB tables will always contain DATA_N_SYS_COLS = 3
columns, 2 or 3 of which are present in the clustered index.
We remove the predicate that was added in MySQL 5.7 as part of WL#7682.
On Microsoft Windows, InnoDB writes the path separator \ to the
redo log file, while on all other platforms, / is being used.
fil_name_parse(): Normalize the parsed path separators to the
native format. This allows backups or data sets to be portable
between Windows and other systems.
MariaDB before MDEV-5800 in version 10.2.2 did not support
indexed virtual columns. Non-persistent virtual columns were
hidden from storage engines. Only starting with MDEV-5800, InnoDB
would create internal metadata on virtual columns.
Similar to what was done in MDEV-18084, MDEV-18090, MDEV-18960, we adjust
one more code path for the old tables.
innobase_build_col_map(): Allocate space for virtual columns in col_map[]
but leave the entries at ULINT_UNDEFINED, noting that the virtual columns
were missing before the table was being rebuilt.
InnoDB includes 3 parsers, which use 3 lexical analyzers that
are generated with flex. Flex versions before 2.6 emitted
the keyword "register", which is deprecated in C++17.
The lexical analyzers were regenerated as follows:
for s in storage/innobase storage/xtradb
do
(cd "$s"/pars; ./make_flex.sh)
touch "$s"/fts/*.l
make -C "$s"/fts -f Makefile.query
done
The MDEV-11738/MDEV-11581 fix was supposed to add the column
ROTATING_OR_FLUSHING to the INFORMATION_SCHEMA table
INNODB_TABLESPACES_ENCRYPTION, but it also added that column to
INNODB_TABLESPACES_SCRUBBING in InnoDB (not XtraDB).
The extra column was never initialized. We will remove it,
because key rotation has nothing to do with the scrubbing of
tablespace data.
This reverts commit 21b2fada7a
and commit 81d71ee6b2.
The MDEV-18464 change introduces a few data race issues. Contrary to
the documentation, the field trx_t::victim is not always being protected
by lock_sys_t::mutex and trx_t::mutex. Most importantly, it seems
that KILL QUERY could wrongly avoid acquiring both mutexes when
invoking lock_trx_handle_wait_low(), in case another thread had
already set trx->victim=true.
We also revert MDEV-12009, because it should depend on the MDEV-18464
fix being present.
1) Avoid writing of MLOG_INDEX_LOAD redo log record during inplace
alter table when the table is empty and also for spatial index.
2) Avoid creation of temporary merge file for spatial index during
index creation process.
Pushed the decision for innodb transaction and system
locking down to lock0lock.cc level. With this,
we can avoid releasing these mutexes for executions
where these mutexes were acquired upfront.
This patch will also fix BF aborting of native threads, e.g.
threads which have declared wsrep_on=OFF. Earlier, we have
used, for innodb trx locks, was_chosen_as_deadlock_victim
flag, for marking inodb transactions, which are victims for
wsrep BF abort. With native threads (wsrep_on==OFF), re-using
was_chosen_as_deadlock_victim flag may lead to inteference
with real deadlock, and to deal with this, the patch has added new
flag for marking wsrep BF aborts only: victim=true
Similar way if replication decides to abort one of the threads
we mark victim by: victim=true
innobase_kill_query
Remove lock sys and trx mutex handling.
wsrep_innobase_kill_one_trx
Mark victim trx with victim=true
trx0trx.h
Remove trx_abort_t type and abort type variable from
trx struct. Add victim variable to trx.
wsrep_kill_victim
Remove abort_type
lock_report_waiters_to_mysql
Take also trx mutex and mark trx as a victim for
replication abort.
lock_trx_handle_wait_low
New low level function to check whether the transaction
has already been rolled back because it was selected as
a deadlock victim, or if it has to wait then cancel
the wait lock.
lock_trx_handle_wait
If transaction is not marked as victim take lock sys
and trx mutex before calling lock_trx_handle_wait_low
and release them after that.
row_search_for_mysql
Remove lock sys and trx mutex taking and releasing.
trx_rollback_to_savepoint_for_mysql_low
trx_commit_in_memory
Clean up victim variable.
now we can afford it. Fix -Werror errors. Note:
* old gcc is bad at detecting uninit variables, disable it.
* time_t is int or long, cast it for printf's
In a Ubuntu Xenial build environment, the compiler identified as
g++-5.real (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
seems to be emitting incorrect code for the compilation unit
trx0rec.cc, triggering a bogus-looking AddressSanitizer report
of an invalid read of something in the function trx_undo_rec_get_pars().
This is potentially affecting any larger tests where the InnoDB
purge subsystem is being exercised.
When the optimization level of trx0rec.cc is limited to -O1, no
bogus failure is being reported. With -O2 or -O3, a lot of things
seemed to be inlined in the function, and the disassembly of the
generated code did not make sense to me.
Before MDEV-12113 (MariaDB Server 10.1.25), on shutdown InnoDB would write
the current LSN to the first page of each file of the system tablespace.
This is incompatible with MariaDB's InnoDB table encryption, because
encryption repurposed the field for an encryption key ID and checksum.
buf_page_is_corrupted(): For the InnoDB system tablespace, skip
FIL_PAGE_FILE_FLUSH_LSN when checking if a page is all zero,
because the first page of each file in the system tablespace can
contain nonzero bytes in the field.
The test case for reproducing MDEV-14126 demonstrates that InnoDB can
end up with an index tree where a non-leaf page has only one child page.
The test case innodb.innodb_bug14676111 demonstrates that such pages
are sometimes unavoidable, because InnoDB does not implement any sort
of B-tree rotation.
But, there is no reason to allow a root page with only one child page.
btr_cur_node_ptr_delete(): Replaces btr_node_ptr_delete().
btr_page_get_father(): Declare globally.
btr_discard_only_page_on_level(): Declare with ATTRIBUTE_COLD.
It turns out that this function is not covered by the
innodb.innodb_bug14676111 test case after all.
btr_discard_page(): If the root page ends up having only one child
page, shrink the tree by invoking btr_lift_page_up().
