Fixes also MDEV-13488: InnoDB writes CRYPT_INFO even though
encryption is not enabled.
Fixes also MDEV-13093: Leak of Datafile::m_crypt_info on
shutdown after failed startup.
Problem was that we created encryption metadata (crypt_data) for
system tablespace even when no encryption was enabled and too early.
System tablespace can be encrypted only using key rotation.
Test innodb-key-rotation-disable, innodb_encryption, innodb_lotoftables
require adjustment because INFORMATION_SCHEMA INNODB_TABLESPACES_ENCRYPTION
contain row only if tablespace really has encryption metadata.
xb_load_single_table_tablespace(): Do not call
fil_space_destroy_crypt_data() any more, because Datafile::m_crypt_data
has been removed.
fil_crypt_realloc_iops(): Avoid divide by zero.
fil_crypt_set_thread_cnt(): Set fil_crypt_threads_event if
encryption threads exist. This is required to find tablespaces
requiring key rotation if no other changes happen.
fil_crypt_find_space_to_rotate(): Decrease the amount of time waiting
when nothing happens to better enable key rotation on startup.
fil_ibd_open(), fil_ibd_load(): Load possible crypt_data from first
page.
class Datafile, class SysTablespace : remove m_crypt_info field.
Datafile::get_first_page(): Return a pointer to first page buffer.
fsp_header_init(): Write encryption metadata to page 0 only if
tablespace is encrypted or encryption is disabled by table option.
i_s_dict_fill_tablespaces_encryption(): Skip tablespaces that do not
contain encryption metadata. This is required to avoid too early
wait condition trigger in encrypted -> unencrypted state transfer.
innodb.table_flags: Adjust the test case. Due to the MDEV-12873 fix
in 10.2, the corrupted flags for table test.td would be converted,
and a tablespace flag mismatch will occur when trying to open the file.
Problem appears to be that the function fsp_flags_try_adjust()
is being unconditionally invoked on every .ibd file on startup.
Based on performance investigation also the top function
fsp_header_get_crypt_offset() needs to addressed.
Ported implementation of fsp_header_get_encryption_offset()
function from 10.2 to fsp_header_get_crypt_offset().
Introduced a new function fil_crypt_read_crypt_data()
to read page 0 if it is not yet read.
fil_crypt_find_space_to_rotate(): Now that page 0 for every .ibd
file is not read on startup we need to check has page 0 read
from space that we investigate for key rotation, if it is not read
we read it.
fil_space_crypt_get_status(): Now that page 0 for every .ibd
file is not read on startup here also we need to read page 0
if it is not yet read it. This is needed
as tests use IS query to wait until background encryption
or decryption has finished and this function is used to
produce results.
fil_crypt_thread(): Add is_stopping condition for tablespace
so that we do not rotate pages if usage of tablespace should
be stopped. This was needed for failure seen on regression
testing.
fil_space_create: Remove page_0_crypt_read and extra
unnecessary info output.
fil_open_single_table_tablespace(): We call fsp_flags_try_adjust
only when when no errors has happened and server was not started
on read only mode and tablespace validation was requested or
flags contain other table options except low order bits to
FSP_FLAGS_POS_PAGE_SSIZE position.
fil_space_t::page_0_crypt_read removed.
Added test case innodb-first-page-read to test startup when
encryption is on and when encryption is off to check that not
for all tables page 0 is read on startup.
The issue was a bad merge of MDEV-12253 from 10.1 to 10.2
in commit f9cc391863.
In that merge, I wrongly assumed that all test file conflicts
for mysql-test/suite/encryption had been properly resolved in
bb-10.2-MDEV-12253 (commit 76aa6be77635c7017459ce33b41c837c9acb606d)
while in fact, some files there had been copied from the 10.1 branch.
This commit is based on a manually done conflict resolution of
the mysql-test/suite/encryption on the same merge, applied to
the current 10.2 branch.
As part of this commit, the test encryption.innodb-bad-key-change4
which was shortly disabled due to MDEV-11336 will be re-enabled again.
(While the test enables innodb_defragment, it does not fail even though
enabling innodb_defragment currently has no effect.)
This only merges MDEV-12253, adapting it to MDEV-12602 which is already
present in 10.2 but not yet in the 10.1 revision that is being merged.
TODO: Error handling in crash recovery needs to be improved.
If a page cannot be decrypted (or read), we should cleanly abort
the startup. If innodb_force_recovery is specified, we should
ignore the problematic page and apply redo log to other pages.
Currently, the test encryption.innodb-redo-badkey randomly fails
like this (the last messages are from cmake -DWITH_ASAN):
2017-05-05 10:19:40 140037071685504 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1635994
2017-05-05 10:19:40 140037071685504 [ERROR] InnoDB: Missing MLOG_FILE_NAME or MLOG_FILE_DELETE before MLOG_CHECKPOINT for tablespace 1
2017-05-05 10:19:40 140037071685504 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[2201] with error Data structure corruption
2017-05-05 10:19:41 140037071685504 [Note] InnoDB: Starting shutdown...
i=================================================================
==5226==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x612000018588 in thread T0
#0 0x736750 in operator delete(void*) (/mariadb/server/build/sql/mysqld+0x736750)
#1 0x1e4833f in LatchCounter::~LatchCounter() /mariadb/server/storage/innobase/include/sync0types.h:599:4
#2 0x1e480b8 in LatchMeta<LatchCounter>::~LatchMeta() /mariadb/server/storage/innobase/include/sync0types.h:786:17
#3 0x1e35509 in sync_latch_meta_destroy() /mariadb/server/storage/innobase/sync/sync0debug.cc:1622:3
#4 0x1e35314 in sync_check_close() /mariadb/server/storage/innobase/sync/sync0debug.cc:1839:2
#5 0x1dfdc18 in innodb_shutdown() /mariadb/server/storage/innobase/srv/srv0start.cc:2888:2
#6 0x197e5e6 in innobase_init(void*) /mariadb/server/storage/innobase/handler/ha_innodb.cc:4475:3
Most notably, this includes MDEV-11623, which includes a fix and
an upgrade procedure for the InnoDB file format incompatibility
that is present in MariaDB Server 10.1.0 through 10.1.20.
In other words, this merge should address
MDEV-11202 InnoDB 10.1 -> 10.2 migration does not work
The function fsp_flags_try_adjust(), which is called on startup,
is incrementing the Innodb_pages0_read counts for every affected file.
Adjust the result of encryption.innodb_lotoftables accordingly.
Perform a slow shutdown at the start of the test, and create all
InnoDB tables with STATS_PERSISTENT=0, so that any I/O related to
background tasks (change buffer merge, purge, persistent statistics)
should be eliminated.
Analysis: By design InnoDB was reading first page of every .ibd file
at startup to find out is tablespace encrypted or not. This is
because tablespace could have been encrypted always, not
encrypted newer or encrypted based on configuration and this
information can be find realible only from first page of .ibd file.
Fix: Do not read first page of every .ibd file at startup. Instead
whenever tablespace is first time accedded we will read the first
page to find necessary information about tablespace encryption
status.
TODO: Add support for SYS_TABLEOPTIONS where all table options
encryption information included will be stored.
