mariadb/mysql-test/suite/encryption/t
Marko Mäkelä 169c00994b MDEV-12699 Improve crash recovery of corrupted data pages
InnoDB crash recovery used to read every data page for which
redo log exists. This is unnecessary for those pages that are
initialized by the redo log. If a newly created page is corrupted,
recovery could unnecessarily fail. It would suffice to reinitialize
the page based on the redo log records.

To add insult to injury, InnoDB crash recovery could hang if it
encountered a corrupted page. We will fix also that problem.
InnoDB would normally refuse to start up if it encounters a
corrupted page on recovery, but that can be overridden by
setting innodb_force_recovery=1.

Data pages are completely initialized by the records
MLOG_INIT_FILE_PAGE2 and MLOG_ZIP_PAGE_COMPRESS.
MariaDB 10.4 additionally recognizes MLOG_INIT_FREE_PAGE,
which notifies that a page has been freed and its contents
can be discarded (filled with zeroes).

The record MLOG_INDEX_LOAD notifies that redo logging has
been re-enabled after being disabled. We can avoid loading
the page if all buffered redo log records predate the
MLOG_INDEX_LOAD record.

For the internal tables of FULLTEXT INDEX, no MLOG_INDEX_LOAD
records were written before commit aa3f7a107c.
Hence, we will skip these optimizations for tables whose
name starts with FTS_.

This is joint work with Thirunarayanan Balathandayuthapani.

fil_space_t::enable_lsn, file_name_t::enable_lsn: The LSN of the
latest recovered MLOG_INDEX_LOAD record for a tablespace.

mlog_init: Page initialization operations discovered during
redo log scanning. FIXME: This really belongs in recv_sys->addr_hash,
and should be removed in MDEV-19176.

recv_addr_state: Add the new state RECV_WILL_NOT_READ to
indicate that according to mlog_init, the page will be
initialized based on redo log record contents.

recv_add_to_hash_table(): Set the RECV_WILL_NOT_READ state
if appropriate. For now, we do not treat MLOG_ZIP_PAGE_COMPRESS
as page initialization. This works around bugs in the crash
recovery of ROW_FORMAT=COMPRESSED tables.

recv_mark_log_index_load(): Process a MLOG_INDEX_LOAD record
by resetting the state to RECV_NOT_PROCESSED and by updating
the fil_name_t::enable_lsn.

recv_init_crash_recovery_spaces(): Copy fil_name_t::enable_lsn
to fil_space_t::enable_lsn.

recv_recover_page(): Add the parameter init_lsn, to ignore
any log records that precede the page initialization.
Add DBUG output about skipped operations.

buf_page_create(): Initialize FIL_PAGE_LSN, so that
recv_recover_page() will not wrongly skip applying
the page-initialization record due to the field containing
some newer LSN as a leftover from a different page.
Do not invoke ibuf_merge_or_delete_for_page() during
crash recovery.

recv_apply_hashed_log_recs(): Remove some unnecessary lookups.
Note if a corrupted page was found during recovery.
After invoking buf_page_create(), do invoke
ibuf_merge_or_delete_for_page() via mlog_init.ibuf_merge()
in the last recovery batch.

ibuf_merge_or_delete_for_page(): Relax a debug assertion.

innobase_start_or_create_for_mysql(): Abort startup if
a corrupted page was found during recovery. Corrupted pages
will not be flagged if innodb_force_recovery is set.
However, the recv_sys->found_corrupt_fs flag can be set
regardless of innodb_force_recovery if file names are found
to be incorrect (for example, multiple files with the same
tablespace ID).
2019-04-17 13:58:41 +03:00
..
aria_tiny.test MDEV-8022 Assertion `rc == 0' failed in ma_encrypt on dropping an encrypted Aria table 2015-05-13 14:27:17 +02:00
corrupted_during_recovery.test MDEV-12699 Improve crash recovery of corrupted data pages 2019-04-17 13:58:41 +03:00
create_or_replace.opt MDEV-8391: encryption.innodb-page_encryption_compression fails 2015-06-29 08:49:58 +03:00
create_or_replace.test Fix bunch of test failures and solaris build missing include. 2016-09-09 10:01:18 +03:00
debug_key_management.opt Test debug_key_management fails sporadically in buildbot. 2015-10-29 10:35:37 +02:00
debug_key_management.test Tests: detect table count for some encryption tests 2018-01-05 10:31:11 +02:00
encrypt_and_grep.opt MDEV-12253: Buffer pool blocks are accessed after they have been freed 2017-04-26 15:19:16 +03:00
encrypt_and_grep.test MDEV-11984 Avoid accessing SYS_TABLESPACES unnecessarily 2018-03-22 18:01:29 +02:00
encryption_force.opt
encryption_force.test MDEV-14219 Allow online table rebuild when encryption or compression parameters change 2017-10-31 09:10:25 +02:00
filekeys-data.enc
filekeys-data.key
filekeys-tooshort.enc MDEV-10888: encryption.filekeys_emptyfile fails in buildbot with valgrind 2016-09-27 07:54:27 +03:00
filekeys_badtest.inc MDEV-12762 Some files in current 10.2 tree seem to be reverted to an old state 2017-05-10 09:06:57 +03:00
filekeys_emptyfile.opt MDEV-9613: keyfile without any keys crashes mysqld on loading file_key_management plugin 2016-03-22 22:20:33 +02:00
filekeys_emptyfile.test MDEV-9613: keyfile without any keys crashes mysqld on loading file_key_management plugin 2016-03-22 22:20:33 +02:00
filekeys_encfile.opt
filekeys_encfile.test
filekeys_encfile_bad.opt
filekeys_encfile_bad.test
filekeys_encfile_badfile.opt
filekeys_encfile_badfile.test
filekeys_encfile_file.opt
filekeys_encfile_file.test
filekeys_encfile_no.opt
filekeys_encfile_no.test
filekeys_goodtest.inc Merge 10.1 into 10.2 2017-11-01 09:55:00 +02:00
filekeys_nofile.test fix the encryption.filekeys_nofile test 2015-10-10 14:19:02 +02:00
filekeys_plugin.inc misc encryption tests fixes 2015-06-27 09:40:54 +02:00
filekeys_plugin.opt
filekeys_plugin_exists.inc Fix test failures seen on buildbot where file_key_management plugin 2015-09-17 16:40:31 +03:00
filekeys_syntax.opt
filekeys_syntax.test file_key_management plugin: complain if key id 1 is not found 2015-06-02 19:00:23 +02:00
filekeys_tooshort.opt MDEV-10888: encryption.filekeys_emptyfile fails in buildbot with valgrind 2016-09-27 07:54:27 +03:00
filekeys_tooshort.test MDEV-10888: encryption.filekeys_emptyfile fails in buildbot with valgrind 2016-09-27 07:54:27 +03:00
filekeys_unencfile.opt
filekeys_unencfile.test
innochecksum.test MDEV-11939: innochecksum mistakes a file for an encrypted one (page 0 invalid) 2017-08-08 09:41:09 +03:00
innodb-bad-key-change.test MDEV-18733 MariaDB slow start after crash recovery 2019-04-03 19:56:03 +03:00
innodb-bad-key-change2.test MDEV-18733 MariaDB slow start after crash recovery 2019-04-03 19:56:03 +03:00
innodb-bad-key-change3.test MDEV-12762 Some files in current 10.2 tree seem to be reverted to an old state 2017-05-10 09:06:57 +03:00
innodb-bad-key-change4.opt MDEV-12253: Buffer pool blocks are accessed after they have been freed 2017-04-26 15:19:16 +03:00
innodb-bad-key-change4.test MDEV-18733 MariaDB slow start after crash recovery 2019-04-03 19:56:03 +03:00
innodb-checksum-algorithm.test Merge 10.1 into 10.2 2018-12-13 15:47:38 +02:00
innodb-compressed-blob.combinations MDEV-12114: install_db shows corruption for rest encryption and innodb_checksum_algorithm=strict_none 2017-06-01 14:07:48 +03:00
innodb-compressed-blob.test MDEV-18733 MariaDB slow start after crash recovery 2019-04-03 19:56:03 +03:00
innodb-discard-import-change.test MDEV-12762 Some files in current 10.2 tree seem to be reverted to an old state 2017-05-10 09:06:57 +03:00
innodb-discard-import.test Merge branch '10.1' into 10.2 2018-06-21 23:47:39 +02:00
innodb-encr-threads.test InnoDB: check if scrubbing is enabled before scanning the tablespace 2015-05-15 18:12:02 +02:00
innodb-encryption-alter.test Merge 10.1 into 10.2 2019-03-04 16:46:58 +02:00
innodb-encryption-disable.test MDEV-18733 MariaDB slow start after crash recovery 2019-04-03 19:56:03 +03:00
innodb-first-page-read.opt MDEV-12610: MariaDB start is slow 2017-06-09 13:15:39 +03:00
innodb-first-page-read.test MDEV-13557: Startup failure, unable to decrypt ibdata1 2017-08-31 08:36:56 +03:00
innodb-force-corrupt.test MDEV-18733 MariaDB slow start after crash recovery 2019-04-03 19:56:03 +03:00
innodb-key-rotation-disable.opt MDEV-11738: Mariadb uses 100% of several of my 8 cpus doing nothing 2017-03-14 16:23:10 +02:00
innodb-key-rotation-disable.test Clean up a test 2017-05-20 08:41:34 +03:00
innodb-missing-key.opt MDEV-11004: Unable to start (Segfault or os error 2) when encryption key missing 2016-10-29 10:09:06 +03:00
innodb-missing-key.test MDEV-18733 MariaDB slow start after crash recovery 2019-04-03 19:56:03 +03:00
innodb-page_encryption-32k.opt
innodb-page_encryption-32k.test MDEV-12762 Some files in current 10.2 tree seem to be reverted to an old state 2017-05-10 09:06:57 +03:00
innodb-page_encryption.test MDEV-12762 Some files in current 10.2 tree seem to be reverted to an old state 2017-05-10 09:06:57 +03:00
innodb-page_encryption_compression.test MDEV-12762 Some files in current 10.2 tree seem to be reverted to an old state 2017-05-10 09:06:57 +03:00
innodb-page_encryption_log_encryption.opt MDEV-8143: InnoDB: Database page corruption on disk or a failed file read 2015-05-14 11:32:24 +03:00
innodb-page_encryption_log_encryption.test MDEV-12762 Some files in current 10.2 tree seem to be reverted to an old state 2017-05-10 09:06:57 +03:00
innodb-read-only.opt MDEV-11835: InnoDB: Failing assertion: free_slot != NULL on 2017-01-19 08:19:08 +02:00
innodb-read-only.test MDEV-11835: InnoDB: Failing assertion: free_slot != NULL on 2017-01-19 08:19:08 +02:00
innodb-redo-badkey.opt MDEV-12253: Buffer pool blocks are accessed after they have been freed 2017-04-26 15:19:16 +03:00
innodb-redo-badkey.test MDEV-13103: Improve 'cannot be decrypted' error message 2018-06-13 16:02:40 +03:00
innodb-redo-nokeys.opt MDEV-12253: Buffer pool blocks are accessed after they have been freed 2017-04-26 15:19:16 +03:00
innodb-redo-nokeys.test Merge branch '10.1' into 10.2 2018-06-21 23:47:39 +02:00
innodb-remove-encryption.test MDEV-15566: System tablespace does not easily key rotate to unencrypted 2018-04-06 12:59:43 +03:00
innodb-scrub-log.opt MDEV-11705: InnoDB: Failing assertion: (&log_sys->mutex)->is_owned() if server started with innodb-scrub-log 2017-01-03 11:22:49 +02:00
innodb-scrub-log.test MDEV-11705: InnoDB: Failing assertion: (&log_sys->mutex)->is_owned() if server started with innodb-scrub-log 2017-01-03 11:22:49 +02:00
innodb-spatial-index.opt MDEV-11974: MariaDB 10.2 encryption does not support spatial indexes 2017-02-08 09:05:15 +02:00
innodb-spatial-index.test cleanup: don't ---replace_regex /#sql-.*/#sql-temporary/ 2019-02-05 01:34:17 +01:00
innodb_encrypt_log.opt The test failed once on Buildbot with the result difference: 2017-08-23 14:40:23 +03:00
innodb_encrypt_log.test Make a test more robust 2017-09-08 09:38:42 +03:00
innodb_encrypt_log_corruption.opt MDEV-11782 WIP: Clean up the code, and add a test. 2017-02-07 11:55:16 +02:00
innodb_encrypt_log_corruption.test MDEV-11782: Redefine the innodb_encrypt_log format 2017-02-15 08:07:20 +02:00
innodb_encryption-page-compression.opt
innodb_encryption-page-compression.test MDEV-14814: encryption.innodb_encryption-page-compression failed in buildbot with timeout on wait condition 2018-02-24 22:39:21 +02:00
innodb_encryption.opt MDEV-9010 Encryption preset file contains different configuration preset then documentation 2015-10-29 10:46:37 +01:00
innodb_encryption.test MDEV-11984 Avoid accessing SYS_TABLESPACES unnecessarily 2018-03-22 18:01:29 +02:00
innodb_encryption_discard_import.opt MDEV-8708: InnoDB temp file encryption 2015-09-08 08:38:12 +03:00
innodb_encryption_discard_import.test Merge 10.1 into 10.2 2018-04-21 11:58:32 +03:00
innodb_encryption_filekeys.opt MDEV-8832: Encryption meta data should not be stored when ENCRYPTED=DEFAULT and innodb-encrypt-tables=OFF 2015-09-23 17:34:36 +03:00
innodb_encryption_filekeys.test MDEV-15672: encryption.innodb_encryption_filekeys - typo in I_S column name: ENCRYPTION_SHCEME 2018-04-13 14:03:12 +03:00
innodb_encryption_is.opt MDEV-9640: Add used key_id to INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION 2016-03-18 11:48:49 +02:00
innodb_encryption_is.test MDEV-12762 Some files in current 10.2 tree seem to be reverted to an old state 2017-05-10 09:06:57 +03:00
innodb_encryption_row_compressed.opt MDEV-9931: InnoDB reads first page of every .ibd file at startup 2016-09-22 16:38:24 +03:00
innodb_encryption_row_compressed.test MDEV-12762 Some files in current 10.2 tree seem to be reverted to an old state 2017-05-10 09:06:57 +03:00
innodb_encryption_tables.opt
innodb_encryption_tables.test MDEV-12762 Some files in current 10.2 tree seem to be reverted to an old state 2017-05-10 09:06:57 +03:00
innodb_first_page.opt
innodb_first_page.test MDEV-12762 Some files in current 10.2 tree seem to be reverted to an old state 2017-05-10 09:06:57 +03:00
innodb_lotoftables.opt MDEV-9931: InnoDB reads first page of every .ibd file at startup 2016-09-22 16:38:24 +03:00
innodb_lotoftables.test MDEV-11531: encryption.innodb_lotoftables failed in buildbot 2018-03-28 13:14:24 +03:00
innodb_onlinealter_encryption.opt MDEV-12253: Buffer pool blocks are accessed after they have been freed 2017-04-26 15:19:16 +03:00
innodb_onlinealter_encryption.test MDEV-12762 Some files in current 10.2 tree seem to be reverted to an old state 2017-05-10 09:06:57 +03:00
innodb_page_encryption_key_change.test MDEV-12762 Some files in current 10.2 tree seem to be reverted to an old state 2017-05-10 09:06:57 +03:00
innodb_scrub.opt MDEV-12762 Some files in current 10.2 tree seem to be reverted to an old state 2017-05-10 09:06:57 +03:00
innodb_scrub.test MDEV-12762 Some files in current 10.2 tree seem to be reverted to an old state 2017-05-10 09:06:57 +03:00
innodb_scrub_background.opt MDEV-12762 Some files in current 10.2 tree seem to be reverted to an old state 2017-05-10 09:06:57 +03:00
innodb_scrub_background.test Correct a wait condition in a disabled test 2018-03-22 18:01:29 +02:00
second_plugin-12863.test MDEV-12863 No table can be created after second encryption plugin attempted to load 2017-08-08 14:52:08 +02:00
tempfiles.combinations MDEV-14868 MariaDB server crashes after using ROLLBACK TO when encrypt_tmp_files=ON 2018-02-08 22:54:37 +01:00
tempfiles.opt fix encryption.tempfiles to check that encrypt_tmp_files is ON 2018-02-08 22:54:37 +01:00
tempfiles.test Merge branch '10.1' into 10.2 2018-02-11 13:47:16 +01:00