Problem is that page 0 and its possible enrryption information
is not read for undo tablespaces.
fil_crypt_get_latest_key_version(): Do not send event to
encryption threads if event does not yet exists. Seen
on regression testing.
fil_read_first_page: Add new parameter does page belong to
undo tablespace and if it does, we do not read FSP_HEADER.
srv_undo_tablespace_open : Read first page of the tablespace
to get crypt_data if it exists and pass it to fil_space_create.
Tested using innodb_encryption with combinations with
innodb-undo-tablespaces.
This bug was a regression caused by MDEV-12698.
On non-leaf pages, the delete-mark flag in the node pointer records is
basically garbage. (Delete-marking only makes sense at the leaf level
anyway. The purpose of the delete-mark is to tell MVCC, locking and purge
that a leaf-level record does not exist in the READ UNCOMMITTED view,
but it used to exist.)
Node pointer records and non-leaf pages are glue that attaches multiple
leaf pages to an index. This glue is supposed to be transparent to the
transactional layer.
When a page is split, InnoDB creates a node pointer record out of the
child page record that the cursor is positioned on. The node pointer record
for the parent page will be a copy of the child page record, amended with
the child page number. If the child page record happened to carry the
delete-mark flag, then the node pointer record would also carry this flag
(even though the flag makes no sense outside child pages).
(On a related note, for the first node pointer record in the first
node pointer page of each tree level, if the MIN_REC_FLAG is set,
the rest of the record contents (except the child page number)
is basically garbage. From this garbage you could deduce at which point
the child was originally split.)
page_scan_method_t: Replace with bool, as there are only 2 values.
dict_stats_scan_page(): Replace the parameter scan_method with is_leaf.
Ignore the bogus (garbage) delete-mark flag if !is_leaf.
# ib_logfile0 expecting FOUND
-FOUND 3 /public|gossip/ in ib_logfile0
+FOUND 2 /public|gossip/ in ib_logfile0
The most plausible explanation for this difference
should be that the redo log payload grew was so big that
one of the strings (for writing the undo log record,
clustered index record, and secondary index record)
was written to ib_logfile1 instead of ib_logfile0.
Let us run the test with --innodb-log-files-in-group=1 so that
only a single log file will be used.
When MySQL 5.0.3 introduced InnoDB support for two-phase commit,
it also introduced the questionable logic to roll back XA PREPARE
transactions on startup when innodb_force_recovery is 1 or 2.
Remove this logic in order to avoid unwanted side effects when
innodb_force_recovery is being set for other reasons. That is,
XA PREPARE transactions will always remain in that state until
InnoDB receives an explicit XA ROLLBACK or XA COMMIT request
from the upper layer.
At the time the logic was introduced in MySQL 5.0.3, there already
was a startup parameter that is the preferred way of achieving
the behaviour: --tc-heuristic-recover=ROLLBACK.
The parameter --innodb-sync-debug, which is disabled by default,
aims to find potential deadlocks in InnoDB.
When the parameter is enabled, lots of tests failed. Most of these
failures were due to bogus diagnostics. But, as part of this fix,
we are also fixing a bug in error handling code and removing dead
code, and fixing cases where an uninitialized mutex was being
locked and unlocked.
dict_create_foreign_constraints_low(): Remove an extraneous
mutex_exit() call that could cause corruption in an error handling
path. Also, do not unnecessarily acquire dict_foreign_err_mutex.
Its only purpose is to control concurrent access to
dict_foreign_err_file.
row_ins_foreign_trx_print(): Replace a redundant condition with a
debug assertion.
srv_dict_tmpfile, srv_dict_tmpfile_mutex: Remove. The
temporary file is never being written to or read from.
log_free_check(): Allow SYNC_FTS_CACHE (fts_cache_t::lock)
to be held.
ha_innobase::inplace_alter_table(), row_merge_insert_index_tuples():
Assert that no unexpected latches are being held.
sync_latch_meta_init(): Properly initialize dict_operation_lock_key
at SYNC_DICT_OPERATION. dict_sys->mutex is SYNC_DICT, and
the now-removed SRV_DICT_TMPFILE was wrongly registered at
SYNC_DICT_OPERATION.
buf_block_init(): Correctly register buf_block_t::debug_latch.
It was previously misleadingly reported as LATCH_ID_DICT_FOREIGN_ERR.
latch_level_t: Correct the relative latching order of
SYNC_IBUF_PESS_INSERT_MUTEX,SYNC_INDEX_TREE and
SYNC_FILE_FORMAT_TAG,SYNC_DICT_OPERATION to avoid bogus failures.
row_drop_table_for_mysql(): Avoid accessing btr_defragment_mutex
if the defragmentation thread has not been started. This is the
case during fts_drop_orphaned_tables() in recv_recovery_rollback_active().
fil_space_destroy_crypt_data(): Avoid acquiring fil_crypt_threads_mutex
when it is uninitialized. We may have created crypt_data before the
mutex was created, and the mutex creation would be skipped if
InnoDB startup failed or --innodb-read-only was specified.
Add suppressions for the read and decompression errors.
This may be 10.3 specific and related to MDEV-13536 which increases
purge activity. But it does not hurt to suppress rarely occurring
and plausible error messages for this fault-injection test already in 10.2.
row_ins_check_foreign_constraint(): On timeout,
return DB_LOCK_WAIT_TIMEOUT instead of DB_LOCK_WAIT,
so that the lock wait will be properly terminated.
Also, replace some redundant assignments.
It looks like this bug was introduced in MySQL 5.7.8 by:
commit a97f6b91227c7e0fc3151cfe5421891e79c12d19
Author: Annamalai Gurusami <annamalai.gurusami@oracle.com>
Date: Tue Jun 9 16:02:31 2015 +0530
Bug #20953265 INNODB: FAILING ASSERTION: RESULT != FTS_INVALID
MDEV-13498 is a performance regression that was introduced in MariaDB 10.2.2
by commit fec844aca8
which introduced some Galera-specific conditions that were being
evaluated even if the write-set replication was not enabled.
MDEV-13246 Stale rows despite ON DELETE CASCADE constraint
is a correctness regression that was introduced by the same commit.
Especially the subcondition
!(parent && que_node_get_type(parent) == QUE_NODE_UPDATE)
which is equivalent to
!parent || que_node_get_type(parent) != QUE_NODE_UPDATE
makes little sense. If parent==NULL, the evaluation would proceed to the
std::find() expression, which would dereference parent. Because no SIGSEGV
was observed related to this, we can conclude that parent!=NULL always
holds. But then, the condition would be equivalent to
que_node_get_type(parent) != QUE_NODE_UPDATE
which would not make sense either, because the std::find() expression
is actually assuming the opposite when casting parent to upd_node_t*.
It looks like this condition never worked properly, or that
it was never properly tested, or both.
wsrep_must_process_fk(): Helper function to check if FOREIGN KEY
constraints need to be processed. Only evaluate the costly std::find()
expression when write-set replication is enabled.
Also, rely on operator<<(std::ostream&, const id_name_t&) and
operator<<(std::ostream&, const table_name_t&) for pretty-printing
index and table names.
row_upd_sec_index_entry(): Add !wsrep_thd_is_BF() to the condition.
This is applying part of "Galera MW-369 FK fixes"
f37b79c6da
that is described by the following part of the commit comment:
additionally: skipping wsrep_row_upd_check_foreign_constraint if thd has
BF, essentially is applier or replaying
This FK check would be needed only for populating parent row FK keys
in write set, so no use for appliers
trx_set_rw_mode(): Check the flag high_level_read_only instead
of testing srv_force_recovery (innodb_force_recovery) directly.
There is no need to prevent the creation of read-write transactions
if innodb_force_recovery=3 is used. Yes, in that mode any recovered
incomplete transactions will not be rolled back, but these transactions
will continue to hold locks on the records that they have modified.
If the new read-write transactions hit conflicts with already existing
(possibly recovered) transactions, the lock wait timeout mechanism
will work just fine.
collateral changes:
* remove a test from innodb_virtual_basic that is already present in
gcol_keys_innodb
* set thd->abort_on_warning for inplace alter, just like it's set
for copy_data_between_tables - to have warnings converted into
errors identically in all alter algorithms
* don't ignore errors in TABLE::update_virtual_field
doing SYSVAR_AUTOSIZE() because of back_log > max_connections
enabled "autosized" flag, and that made IS_SYSVAR_AUTOSIZE()
true, which triggered the second SYSVAR_AUTOSIZE.
Remove back_log <= max_connections limit, back_log
doesn't *always* have to be smaller than max_connections.
SQL Standard behavior for DROP COLUMN xxx RESTRICT:
* If a constraint (UNIQUE or CHECK) uses only the dropped column,
it's automatically dropped too. If it uses many columns - an error.
Only a relevant subset of the InnoDB changes was merged.
In particular, two follow-up bug fixes for the bugs that
were introduced in 5.7.18 but not MariaDB 10.2.7 were omitted.
Because MariaDB 10.2.7 omitted the risky change
Bug#23481444 OPTIMISER CALL ROW_SEARCH_MVCC() AND READ THE INDEX
APPLIED BY UNCOMMITTED ROWS
we do not need the follow-up fixes that were introduced in
MySQL 5.6.37 and MySQL 5.7.19:
Bug#25175249 ASSERTION: (TEMPL->IS_VIRTUAL && !FIELD) || ...
Bug#25793677 INNODB: FAILING ASSERTION: CLUST_TEMPL_FOR_SEC || LEN
The test is for a bug that was introduced in MySQL 5.7.18
but not MariaDB 10.2, because MariaDB did not merge the change
that was considered incomplete and too risky for a GA release:
Bug#23481444 OPTIMISER CALL ROW_SEARCH_MVCC() AND READ THE INDEX
APPLIED BY UNCOMMITTED ROWS
So, we are only merging the test changes from the bug fix in MySQL 5.7.19,
not any code changes:
commit 4f86aca37d551cc756d9187ec901f8c4a68a0543
Author: Thirunarayanan Balathandayuthapani <thirunarayanan.balathandayuth@oracle.com>
Date: Wed Apr 26 11:10:41 2017 +0530
Bug #25793677 INNODB: FAILING ASSERTION: CLUST_TEMPL_FOR_SEC || LEN
Cherry-pick the commit from MySQL 5.7.19, and adapt the test case:
commit 45c933ac19c73a3e9c756a87ee1ba18ba1ac564c
Author: Aakanksha Verma <aakanksha.verma@oracle.com>
Date: Tue Mar 21 10:31:43 2017 +0530
Bug #25189192 ERRORS WHEN RESTARTING MYSQL AFTER RENAME TABLE.
PROBLEM
While renaming table innodb doesn't update the InnoDB Dictionary table
INNODB_SYS_DATAFILES incase there is change in database while doing
rename table. Hence on a restart the server log shows error that it
couldnt find table with old path before rename which has actually been
renamed. So the errors would only vanish if we update the system
tablespace
FIX
Update the innodb dictionary table with new path in the case there is
not a change in the table but the database holding the table as well.
Reviewed-by: Jimmy Yang<Jimmy.Yang@oracle.com>
RB: 15751
Disable change buffering, so that some data that was previously
written to the encrypted redo log will not end up being copied
to the unencrypted redo log due to change buffer merge.
NET can only store current_thd if this NET (or its MYSQL) is not
moved between threads. In FederatedX MYSQL is part of the TABLE,
and a TABLE can migrate between threads.
Fix: associate NET with THD in txn->acquire() , and dissociate
in txn->release()
The file wait_innodb_all_purged.inc waited for InnoDB purge in a way
that only worked in debug builds. The file wait_all_purged.inc
provides a better mechanism.
Always read full page 0 to determine does tablespace contain
encryption metadata. Tablespaces that are page compressed or
page compressed and encrypted do not compare checksum as
it does not exists. For encrypted tables use checksum
verification written for encrypted tables and normal tables
use normal method.
buf_page_is_checksum_valid_crc32
buf_page_is_checksum_valid_innodb
buf_page_is_checksum_valid_none
Modify Innochecksum logging to file to avoid compilation
warnings.
fil0crypt.cc fil0crypt.h
Modify to be able to use in innochecksum compilation and
move fil_space_verify_crypt_checksum to end of the file.
Add innochecksum logging to file.
univ.i
Add innochecksum strict_verify, log_file and cur_page_num
variables as extern.
page_zip_verify_checksum
Add innochecksum logging to file and remove unnecessary code.
innochecksum.cc
Lot of changes most notable able to read encryption
metadata from page 0 of the tablespace.
Added test case where we corrupt intentionally
FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION (encryption key version)
FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION+4 (post encryption checksum)
FIL_DATA+10 (data)
Problem was introduced with the InnoDB 5.7 merge, the code related to
avoiding extra fsync at the end of commit when binlog is enabled. The
MariaDB method for this was removed, but the replacement MySQL method
based on thd_get_durability_property() is not functional in MariaDB.
This commit reverts the offending parts of the merge and adds a test
case, to fix the problem for InnoDB. But other storage engines are
likely to have a similar problem.
This is basically port of WL6045:Improve Innochecksum with some
code refactoring on innochecksum.
Added page0size.h include from 10.2 to make 10.1 vrs 10.2 innochecksum
as identical as possible.
Added page 0 checksum checking and if that fails whole test fails.
Always read full page 0 to determine does tablespace contain
encryption metadata. Tablespaces that are page compressed or
page compressed and encrypted do not compare checksum as
it does not exists. For encrypted tables use checksum
verification written for encrypted tables and normal tables
use normal method.
buf_page_is_checksum_valid_crc32
buf_page_is_checksum_valid_innodb
buf_page_is_checksum_valid_none
Add Innochecksum logging to file
buf_page_is_corrupted
Remove ib_logf and page_warn_strict_checksum
calls in innochecksum compilation. Add innochecksum
logging to file.
fil0crypt.cc fil0crypt.h
Modify to be able to use in innochecksum compilation and
move fil_space_verify_crypt_checksum to end of the file.
Add innochecksum logging to file.
univ.i
Add innochecksum strict_verify, log_file and cur_page_num
variables as extern.
page_zip_verify_checksum
Add innochecksum logging to file.
innochecksum.cc
Lot of changes most notable able to read encryption
metadata from page 0 of the tablespace.
Added test case where we corrupt intentionally
FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION (encryption key version)
FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION+4 (post encryption checksum)
FIL_DATA+10 (data)
Crashes with innodb_page_size=64K. Does not crash at <= 32K.
Problem was that when blob record that was earlier < 16k is
enlarged at update wo that length > 16K it should be stored
externally. However, that was not enforced when page-size = 64K
(note that 16K+1 < 64K/2 i.e. half of the btree leaf page).
btr_cur_optimistic_update: limit max record size to 16K
or in REDUNDANT row format to 16K-1.
