Analysis:
-- InnoDB has n (>0) redo-log files.
-- In the first page of redo-log there is 2 checkpoint records on fixed location (checkpoint is not encrypted)
-- On every checkpoint record there is up to 5 crypt_keys containing the keys used for encryption/decryption
-- On crash recovery we read all checkpoints on every file
-- Recovery starts by reading from the latest checkpoint forward
-- Problem is that latest checkpoint might not always contain the key we need to decrypt all the
redo-log blocks (see MDEV-9422 for one example)
-- Furthermore, there is no way to identify is the log block corrupted or encrypted
For example checkpoint can contain following keys :
write chk: 4 [ chk key ]: [ 5 1 ] [ 4 1 ] [ 3 1 ] [ 2 1 ] [ 1 1 ]
so over time we could have a checkpoint
write chk: 13 [ chk key ]: [ 14 1 ] [ 13 1 ] [ 12 1 ] [ 11 1 ] [ 10 1 ]
killall -9 mysqld causes crash recovery and on crash recovery we read as
many checkpoints as there is log files, e.g.
read [ chk key ]: [ 13 1 ] [ 12 1 ] [ 11 1 ] [ 10 1 ] [ 9 1 ]
read [ chk key ]: [ 14 1 ] [ 13 1 ] [ 12 1 ] [ 11 1 ] [ 10 1 ] [ 9 1 ]
This is problematic, as we could still scan log blocks e.g. from checkpoint 4 and we do
not know anymore the correct key.
CRYPT INFO: for checkpoint 14 search 4
CRYPT INFO: for checkpoint 13 search 4
CRYPT INFO: for checkpoint 12 search 4
CRYPT INFO: for checkpoint 11 search 4
CRYPT INFO: for checkpoint 10 search 4
CRYPT INFO: for checkpoint 9 search 4 (NOTE: NOT FOUND)
For every checkpoint, code generated a new encrypted key based on key
from encryption plugin and random numbers. Only random numbers are
stored on checkpoint.
Fix: Generate only one key for every log file. If checkpoint contains only
one key, use that key to encrypt/decrypt all log blocks. If checkpoint
contains more than one key (this is case for databases created
using MariaDB server version 10.1.0 - 10.1.12 if log encryption was
used). If looked checkpoint_no is found from keys on checkpoint we use
that key to decrypt the log block. For encryption we use always the
first key. If the looked checkpoint_no is not found from keys on checkpoint
we use the first key.
Modified code also so that if log is not encrypted, we do not generate
any empty keys. If we have a log block and no keys is found from
checkpoint we assume that log block is unencrypted. Log corruption or
missing keys is found by comparing log block checksums. If we have
a keys but current log block checksum is correct we again assume
log block to be unencrypted. This is because current implementation
stores checksum only before encryption and new checksum after
encryption but before disk write is not stored anywhere.
It could have happened that one of previous tests already executed
buffer pool dump and set the status variable value, so when it's been
checked, the check passes too early, before the dump starts and
the dump file is created. See more detailed explanation in MDEV-9713.
Fixed by waiting for the current time to change in case it equals
to the timestamp in the status variable, and then checking that
the status variable not only matches the expected pattern, but also
differs from the previous value, whatever it was.
In row_search_for_mysql function on XtraDB there was a old logic
where null bytes were inited. This caused server to think that
key value is null and continue on incorrect path.
FULL
Bug#21753696: MAKE SHOW SLAVE STATUS NON BLOCKING IF IO
THREAD WAITS FOR DISK SPACE
Problem:
========
Currently SHOW SLAVE STATUS blocks if IO thread waits for
disk space. This makes automation tools verifying
server health block on taking relevant action. Finally this
will create SHOW SLAVE STATUS piles.
Analysis:
=========
SHOW SLAVE STATUS hangs on mi->data_lock if relay log write
is waiting for free disk space while holding mi->data_lock.
mi->data_lock is needed to protect the format description
event (mi->format_description_event) which is accessed by
the clients running FLUSH LOGS and slave IO thread. Note
relay log writes don't need to be protected by
mi->data_lock, LOCK_log is used to protect relay log between
IO and SQL thread (see MYSQL_BIN_LOG::append_event). The
code takes mi->data_lock to protect
mi->format_description_event during relay log rotate which
might get triggered right after relay log write.
Fix:
====
Release the data_lock just for the duration of writing into
relay log.
Made change to ensure the following lock order is maintained
to avoid deadlocks.
data_lock, LOCK_log
data_lock is held during relay log rotations to protect
the description event.
The main.merge test case was failing when tested using row based
binlog format.
While analyzing the issue it was found the following issues:
a) The server is calling binlog related code even when a statement will
not be binlogged;
b) The child table list was not present into table structure by the time
to generate the create table statement;
c) The tables in the child table list will not be opened yet when
generating table create info using row based replication;
d) CREATE TABLE LIKE TEMP_TABLE does not preserve original table storage
engine when using row based replication;
This patch addressed all above issues.
@ sql/sql_class.h
Added a function to determine if the binary log is disabled to
the current session. This is related with issue (a) above.
@ sql/sql_table.cc
Added code to skip binary logging related code if the statement
will not be binlogged. This is related with issue (a) above.
Added code to add the children to the query list of the table that
will have its CREATE TABLE generated. This is related with issue (b)
above.
Added code to force the storage engine to be generated into the
CREATE TABLE. This is related with issue (d) above.
@ storage/myisammrg/ha_myisammrg.cc
Added a test to skip a table getting info about a child table if the
child table is not opened. This is related to issue (c) above.
The test created a file in location relative to the datadir
(a few levels above datadir).
The file was created by MariaDB server (via INTO OUTFILE), and
later removed by mysqltest (via remove_file). The problem is that
when the vardir is a symlink, MariaDB server and mysqltest can
resolve such paths differently. MariaDB server would return back
to where the symlink is located, while mysqltest would go above
the real directory. For example, if the test is run with --mem,
and /bld/5.5/mysql-test/var points at /dev/shm/var_auto_X, then
SELECT INTO OUTFILE created a file in /bld/5.5/mysql-test , but
remove_file would look for it in /dev/shm/.
The test is re-written so that all paths are resolved in perl,
the logic itself hasn't changed.
The reason for the assertion failure is that the update statement for
the minimal row image sets only the PK column in the write_set of the
table to true. On the other hand, the trigger aims to update a different
column.
Make sure that triggers update the used columns accordingly, when being
processed.
when replicating old temporal type fields (that don't store
metadata in the binlog), take the precision from
destination fields.
(this fixes the replication failure, crashes were
fixed in a different commit)
* Total order isolation was started twice for FLUSH TABLES, from
reload_acl_and_cache() and from mysql_execute_command(). Removed
the reload_acl_and_cache() part.
* Removed PXC specific stuff from MTR tests
- Eliminates code duplication in query cache patch
- Reduces the number of iterations in mysql-wsrep#201.test
to shorten the execution time
- Adds a new test case that exercises more scenarios
The admin commands in question are:
> OPTIMIZE
> REPAIR
> ANALYZE
For LOCAL or NO_WRITE_TO_BINLOG invocations of these commands, ie
OPTIMIZE LOCAL TABLE <t1>
they are not binlogged as expected.
Also, in addition, they are not executed under TOI.
Hence, they are not propagated to other nodes.
The effect is same as that of wsrep_on=0.
Also added tests for this.
A WSREP_DEBUG for wsrep_register_hton has also been added.
The galera_flush_local test has also been updated for verifying that effects
of NO_WRITE_TO_BINLOG / LOCAL are equivalent to wsrep_on=0 from wsrep
perspective.
(cherry picked from commit 5065122f94a8002d4da231528a46f8d9ddbffdc2)
Conflicts:
sql/sql_admin.cc
sql/sql_reload.cc
sql/wsrep_hton.cc
- Fixes query cache so that it is aware of wsrep_sync_wait.
Query cache would return (possibly stale) results to the
client, regardless of the value of wsrep_sync_wait.
- Includes the test case that reproduced the issue.
- Removes useless call to wsrep_xid_init() in wsrep_apply_events().
Transaction's xid is already initialized at that point.
- Adds call to wsrep_set_SE_checkpoint() for committing TOI events
in the applier side.
- Includes test case that reproduced the issue.
There was two problems. Firstly, if page in ibuf is encrypted but
decrypt failed we should not allow InnoDB to start because
this means that system tablespace is encrypted and not usable.
Secondly, if page decrypt is detected we should return false
from buf_page_decrypt_after_read.
