After-review changes.
For this patch in 10.0, we do not introduce a new public storage engine API,
we just fix the InnoDB/XtraDB issues. In 10.1, we will make a better public
API that can be used for all storage engines (MDEV-6429).
Eliminate the background thread that did deadlock kills asynchroneously.
Instead, we ensure that the InnoDB/XtraDB code can handle doing the kill from
inside the deadlock detection code (when thd_report_wait_for() needs to kill a
later thread to resolve a deadlock).
(We preserve the part of the original patch that introduces dedicated mutex
and condition for the slave init thread, to remove the abuse of
LOCK_thread_count for start/stop synchronisation of the slave init thread).
replication causing replication to fail.
Remove the temporary fix for MDEV-5914, which used READ COMMITTED for parallel
replication worker threads. Replace it with a better, more selective solution.
The issue is with certain edge cases of InnoDB gap locks, for example between
INSERT and ranged DELETE. It is possible for the gap lock set by the DELETE to
block the INSERT, if the DELETE runs first, while the record lock set by
INSERT does not block the DELETE, if the INSERT runs first. This can cause a
conflict between the two in parallel replication on the slave even though they
ran without conflicts on the master.
With this patch, InnoDB will ask the server layer about the two involved
transactions before blocking on a gap lock. If the server layer tells InnoDB
that the transactions are already fixed wrt. commit order, as they are in
parallel replication, InnoDB will ignore the gap lock and allow the two
transactions to proceed in parallel, avoiding the conflict.
Improve the fix for MDEV-6020. When InnoDB itself detects a deadlock, it now
asks the server layer for any preferences about which transaction to roll
back. In case of parallel replication with two transactions T1 and T2 fixed to
commit T1 before T2, the server layer will ask InnoDB to roll back T2 as the
deadlock victim, not T1. This helps in some cases to avoid excessive deadlock
rollback, as T2 will in any case need to wait for T1 to complete before it can
itself commit.
Also some misc. fixes found during development and testing:
- Remove thd_rpl_is_parallel(), it is not used or needed.
- Use KILL_CONNECTION instead of KILL_QUERY when a parallel replication
worker thread is killed to resolve a deadlock with fixed commit
ordering. There are some cases, eg. in sql/sql_parse.cc, where a KILL_QUERY
can be ignored if the query otherwise completed successfully, and this
could cause the deadlock kill to be lost, so that the deadlock was not
correctly resolved.
- Fix random test failure due to missing wait_for_binlog_checkpoint.inc.
- Make sure that deadlock or other temporary errors during parallel
replication are not printed to the the error log; there were some places
around the replication code with extra error logging. These conditions can
occur occasionally and are handled automatically without breaking
replication, so they should not pollute the error log.
- Fix handling of rgi->gtid_sub_id. We need to be able to access this also at
the end of a transaction, to be able to detect and resolve deadlocks due to
commit ordering. But this value was also used as a flag to mark whether
record_gtid() had been called, by being set to zero, losing the value. Now,
introduce a separate flag rgi->gtid_pending, so rgi->gtid_sub_id remains
valid for the entire duration of the transaction.
- Fix one place where the code to handle ignored errors called reset_killed()
unconditionally, even if no error was caught that should be ignored. This
could cause loss of a deadlock kill signal, breaking deadlock detection and
resolution.
- Fix a couple of missing mysql_reset_thd_for_next_command(). This could
cause a prior error condition to remain for the next event executed,
causing assertions about errors already being set and possibly giving
incorrect error handling for following event executions.
- Fix code that cleared thd->rgi_slave in the parallel replication worker
threads after each event execution; this caused the deadlock detection and
handling code to not be able to correctly process the associated
transactions as belonging to replication worker threads.
- Remove useless error code in slave_background_kill_request().
- Fix bug where wfc->wakeup_error was not cleared at
wait_for_commit::unregister_wait_for_prior_commit(). This could cause the
error condition to wrongly propagate to a later wait_for_prior_commit(),
causing spurious ER_PRIOR_COMMIT_FAILED errors.
- Do not put the binlog background thread into the processlist. It causes
too many result differences in mtr, but also it probably is not useful
for users to pollute the process list with a system thread that does not
really perform any user-visible tasks...
Update InnoDB to 5.6.14
Apply MySQL-5.6 hack for MySQL Bug#16434374
Move Aria-only HA_RTREE_INDEX from my_base.h to maria_def.h (breaks an assert in InnoDB)
Fix InnoDB memory leak
This is now otherwise on level wsrep-25.9, but storage/innobase has not been fully merged
wsrep-5.5 is not good source for that, so we probably have to cherry pick innodb changes from wsrep-5.6
Analysis: There is debug assertion ut_ad(trx->start_file != 0); and ut_ad(trx->start_line != 0); on trx_start_low funcition at trx0trx.cc. These fields are initialized on include/trx0trx.h at function trx_start_if_not_started_xa. Thus at trx_prepare_for_mysql function should call trx_start_if_not_started_xa(trx); not trx_start_if_not_started_xa_low(trx) directly;
trx_undo_prev_version_build() should confirm existence of inherited (not-own) external pages.
Bug #14676084 : ROW_UNDO_MOD_UPD_DEL_SEC() DOESN'T NEED UNDO_ROW AND UNDO_EXT INITIALIZED
mtr script could hit the assertion error !bpage->file_page_was_freed using this path.
So, also fixed
rb://1337 approved by Marko Makela.
btr_lift_page_up() writes wrong page number (different by -1) for upper than father page.
But in almost all of the cases, the father page should be root page, no upper
pages. It is very rare path.
In addition the leaf page should not be lifted unless the father page is root.
Because the branch pages should not become the leaf pages.
rb://1336 approved by Marko Makela.
btr_lift_page_up() writes wrong page number (different by -1) for upper than father page.
But in almost all of the cases, the father page should be root page, no upper
pages. It is very rare path.
In addition the leaf page should not be lifted unless the father page is root.
Because the branch pages should not become the leaf pages.
rb://1336 approved by Marko Makela.
Introduce a new storage engine API method commit_checkpoint_request().
This is used to replace the fsync() at the end of every storage engine
commit with a single fsync() when a binlog is rotated.
Binlog rotation is now done during group commit instead of being
delayed until unlog(), removing some server stall and avoiding an
expensive lock/unlock of LOCK_log inside unlog().
Changes in the InnoDB codebase required to compile and
integrate the MEB codebase with MySQL 5.5.
@ storage/innobase/btr/btr0btr.c
Excluded buffer pool usage from MEB build.
buf_pool_from_bpage calls are in buf0buf.ic, and
the buffer pool functions from that file are
disabled in MEB.
@ storage/innobase/buf/buf0buf.c
Disabling more buffer pool functions unused in MEB.
@ storage/innobase/dict/dict0dict.c
Disabling dict_ind_free that is unused in MEB.
@ storage/innobase/dict/dict0mem.c
The include
#include "ha_prototypes.h"
Was causing conflicts with definitions in my_global.h
Linking C executable mysqlbackup
libinnodb.a(dict0mem.c.o): In function `dict_mem_foreign_table_name_lookup_set':
dict0mem.c:(.text+0x91c): undefined reference to `innobase_get_lower_case_table_names'
libinnodb.a(dict0mem.c.o): In function `dict_mem_referenced_table_name_lookup_set':
dict0mem.c:(.text+0x9fc): undefined reference to `innobase_get_lower_case_table_names'
libinnodb.a(dict0mem.c.o): In function `dict_mem_foreign_table_name_lookup_set':
dict0mem.c:(.text+0x96e): undefined reference to `innobase_casedn_str'
libinnodb.a(dict0mem.c.o): In function `dict_mem_referenced_table_name_lookup_set':
dict0mem.c:(.text+0xa4e): undefined reference to `innobase_casedn_str'
collect2: ld returned 1 exit status
make[2]: *** [mysqlbackup] Error 1
innobase_get_lower_case_table_names
innobase_casedn_str
are functions that are part of ha_innodb.cc that is not part of the build
dict_mem_foreign_table_name_lookup_set
function is not there in the current codebase, meaning we do not use it in MEB.
@ storage/innobase/fil/fil0fil.c
The srv_fast_shutdown variable is declared in
srv0srv.c that is not compiled in the
mysqlbackup codebase.
This throws an undeclared error.
From the Manual
---------------
innodb_fast_shutdown
--------------------
The InnoDB shutdown mode. The default value is 1
as of MySQL 3.23.50, which causes a “fast� shutdown
(the normal type of shutdown). If the value is 0,
InnoDB does a full purge and an insert buffer merge
before a shutdown. These operations can take minutes,
or even hours in extreme cases. If the value is 1,
InnoDB skips these operations at shutdown.
This ideally does not matter from mysqlbackup
@ storage/innobase/ha/ha0ha.c
In file included from /home/narayanan/mysql-server/mysql-5.5-meb-rel3.8-innodb-integration-1/storage/innobase/ha/ha0ha.c:34:0:
/home/narayanan/mysql-server/mysql-5.5-meb-rel3.8-innodb-integration-1/storage/innobase/include/btr0sea.h:286:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
make[2]: *** [CMakeFiles/innodb.dir/home/narayanan/mysql-server/mysql-5.5-meb-rel3.8-innodb-integration-1/storage/innobase/ha/ha0ha.c.o] Error 1
make[1]: *** [CMakeFiles/innodb.dir/all] Error 2
make: *** [all] Error 2
# include "sync0rw.h" is excluded from hotbackup compilation in dict0dict.h
This causes extern rw_lock_t* btr_search_latch_temp; to throw a failure because
the definition of rw_lock_t is not found.
@ storage/innobase/include/buf0buf.h
Excluding buffer pool functions that are unused from the
MEB codebase.
@ storage/innobase/include/buf0buf.ic
replicated the exclusion of
#include "buf0flu.h"
#include "buf0lru.h"
#include "buf0rea.h"
by looking at the current codebase in <meb-trunk>/src/innodb
@ storage/innobase/include/dict0dict.h
dict_table_x_lock_indexes, dict_table_x_unlock_indexes, dict_table_is_corrupted,
dict_index_is_corrupted, buf_block_buf_fix_inc_func are unused in MEB and was
leading to compilation errors and hence excluded.
@ storage/innobase/include/dict0dict.ic
dict_table_x_lock_indexes, dict_table_x_unlock_indexes, dict_table_is_corrupted,
dict_index_is_corrupted, buf_block_buf_fix_inc_func are unused in MEB and was
leading to compilation errors and hence excluded.
@ storage/innobase/include/log0log.h
/home/narayanan/mysql-server/mysql-5.5-meb-rel3.8-innodb-integration-1/storage/innobase/include/log0log.h: At top level:
/home/narayanan/mysql-server/mysql-5.5-meb-rel3.8-innodb-integration-1/storage/innobase/include/log0log.h:767:2: error: expected specifier-qualifier-list before â⠂¬Ëœmutex_t’
mutex_t definitions were excluded as seen from ambient code
hence excluding definition for log_flush_order_mutex also.
@ storage/innobase/include/os0file.h
Bug in InnoDB code, create_mode should have been create.
@ storage/innobase/include/srv0srv.h
In file included from /home/narayanan/mysql-server/mysql-5.5-meb-rel3.8-innodb-integration-1/storage/innobase/buf/buf0buf.c:50:0:
/home/narayanan/mysql-server/mysql-5.5-meb-rel3.8-innodb-integration-1/storage/innobase/include/srv0srv.h: At top level:
/home/narayanan/mysql-server/mysql-5.5-meb-rel3.8-innodb-integration-1/storage/innobase/include/srv0srv.h:120:16: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘srv_use_native_aio’
srv_use_native_aio - we do not use native aio of the OS anyway from MEB. MEB does not compile
InnoDB with this option. Hence disabling it.
@ storage/innobase/include/trx0sys.h
[ 56%] Building C object CMakeFiles/innodb.dir/home/narayanan/mysql-server/mysql-5.5-meb-rel3.8-innodb-integration-1/storage/innobase/trx/trx0sys.c.o
/home/narayanan/mysql-server/mysql-5.5-meb-rel3.8-innodb-integration-1/storage/innobase/trx/trx0sys.c: In function ‘trx_sys_read_file_format_id’:
/home/narayanan/mysql-server/mysql-5.5-meb-rel3.8-innodb-integration-1/storage/innobase/trx/trx0sys.c:1499:20: error: ‘TRX_SYS_FILE_FORMAT_TAG_MAGIC_N’ undeclared (first use in this function)
/home/narayanan/mysql-server/mysql-5.5-meb-rel3.8-innodb-integration-1/storage/innobase/trx/trx0sys.c:1499:20: note: each undeclared identifier is reported only once for each function it appears in
/home/narayanan/mysql-server/mysql-5.5-meb-rel3.8-innodb-integration-1/storage/innobase/trx/trx0sys.c: At top level:
/home/narayanan/mysql-server/mysql-5.5-meb-rel3.8-innodb-integration-1/storage/innobase/include/buf0buf.h:607:1: warning: ‘buf_block_buf_fix_inc_func’ declared ‘static’ but never defined
make[2]: *** [CMakeFiles/innodb.dir/home/narayanan/mysql-server/mysql-5.5-meb-rel3.8-innodb-integration-1/storage/innobase/trx/trx0sys.c.o] Error 1
unused calls excluded to enable compilation
@ storage/innobase/mem/mem0dbg.c
excluding #include "ha_prototypes.h" that lead to definitions in ha_innodb.cc
@ storage/innobase/os/os0file.c
InnoDB not compiled with aio support from MEB anyway. Hence excluding this from
the compilation.
@ storage/innobase/page/page0zip.c
page0zip.c:(.text+0x4e9e): undefined reference to `buf_pool_from_block'
collect2: ld returned 1 exit status
buf_pool_from_block defined in buf0buf.ic, most of the file is excluded for compilation of MEB
@ storage/innobase/ut/ut0dbg.c
excluding #include "ha_prototypes.h" since it leads to definitions in ha_innodb.cc
innobase_basename(file) is defined in ha_innodb.cc. Hence excluding that also.
@ storage/innobase/ut/ut0ut.c
cal_tm unused from MEB, was leading to earnings, hence disabling for MEB.
mysql-test/suite/innodb/t/group_commit_crash.test:
remove autoincrement to avoid rbr being used for insert ... select
mysql-test/suite/innodb/t/group_commit_crash_no_optimize_thread.test:
remove autoincrement to avoid rbr being used for insert ... select
mysys/my_addr_resolve.c:
a pointer to a buffer is returned to the caller -> the buffer cannot be on the stack
mysys/stacktrace.c:
my_vsnprintf() is ok here, in 5.5
print page dump
buf_page_print(): Remove the ut_ad(0) from the beginning. Add two flags
(enum buf_page_print_flags) that can be bitwise-ORed together:
BUF_PAGE_PRINT_NO_CRASH:
Do not crash debug builds at the end of buf_page_print().
BUF_PAGE_PRINT_NO_FULL:
Do not print the full page dump. This can be useful when adding
diagnostic printout to flushing or to the doublewrite buffer.
trx_sys_doublewrite_init_or_restore_page(): Replace exit(1) with ut_error,
so that we can get a core dump if this extraordinary condition happens.
rb:924 approved by Sunny Bains
This fix does not remove the underlying cause of the assertion
failure. It just works around the problem, allowing a corrupted
secondary index to be fixed by DROP INDEX and CREATE INDEX (or in the
worst case, by re-creating the table).
ibuf_delete(): If the record to be purged is the last one in the page
or it is not delete-marked, refuse to purge it. Instead, write an
error message to the error log and let a debug assertion fail.
ibuf_set_del_mark(): If the record to be delete-marked is not found,
display some more information in the error log and let a debug
assertion fail.
row_undo_mod_del_unmark_sec_and_undo_update(),
row_upd_sec_index_entry(): Let a debug assertion fail when the record
to be delete-marked is not found.
buf_page_print(): Add ut_ad(0) so that corruption will be more
prominent in stress testing with debug binaries. Add ut_ad(0) here and
there where corruption is noticed.
btr_corruption_report(): Display some data on page_is_comp() mismatch.
btr_assert_not_corrupted(): A wrapper around btr_corruption_report().
Assert that page_is_comp() agrees with the table flags.
rb:911 approved by Inaam Rana
If we meet DB_TOO_MANY_CONCURRENT_TRXS during the execution tab_create_graph from row_create_table_for_mysql(), .ibd file for the table should be created already but was not deleted for the error handling.
rb:875 approved by Jimmy Yang
If we meet DB_TOO_MANY_CONCURRENT_TRXS during the execution tab_create_graph from row_create_table_for_mysql(), .ibd file for the table should be created already but was not deleted for the error handling.
rb:875 approved by Jimmy Yang
This was an attempt to address problems with the Bug#12612184 fix.
Even with this follow-up fix, crash recovery can be broken.
Let us fix the bug later.
