Actual error number returned from the query depends what point
corrupted page is accessed, is it accessed when we read
one of the pages for result set or is it accessed during
background page read.
JOIN_TAB::remove_redundant_bnl_scan_conds() removes select_cond
from a JOIN_TAB if join cache is enabled, and tab->cache_select->cond
is the equal to tab->select_cond.
But after 8d99166c69 the code to initialize join cache was moved
to happen much later than JOIN_TAB::remove_redundant_bnl_scan_conds(),
and that code might, under certain conditions, revert to *not* using
join cache (set_join_cache_denial()).
If JOIN_TAB::remove_redundant_bnl_scan_conds() removes the WHERE
condition from the JOIN_TAB and later set_join_cache_denial() disables
join cache, we end up with no WHERE condition at all.
Fix: move JOIN_TAB::remove_redundant_bnl_scan_conds() to happen
after all possible set_join_cache_denial() calls.
table_already_fk_prelocked() was looking for a table in the wrong
list (not the complete list of prelocked tables, but only in its tail,
starting from the current table - which is always empty for the last
added table), so for circular FKs it kept adding same tables to the list
indefinitely.
Also, include fixes by Vladislav Vaintroub to the
aws_key_management plugin. The AWS C++ SDK specifically depends on
OPENSSL_LIBRARIES, not generic SSL_LIBRARIES (such as YaSSL).
During merge of innodb code QC invalidation was removed from innodb part but not added to server part.
We decided to keep it in innodb to keep server/engine interface the same.
This only merges MDEV-12253, adapting it to MDEV-12602 which is already
present in 10.2 but not yet in the 10.1 revision that is being merged.
TODO: Error handling in crash recovery needs to be improved.
If a page cannot be decrypted (or read), we should cleanly abort
the startup. If innodb_force_recovery is specified, we should
ignore the problematic page and apply redo log to other pages.
Currently, the test encryption.innodb-redo-badkey randomly fails
like this (the last messages are from cmake -DWITH_ASAN):
2017-05-05 10:19:40 140037071685504 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1635994
2017-05-05 10:19:40 140037071685504 [ERROR] InnoDB: Missing MLOG_FILE_NAME or MLOG_FILE_DELETE before MLOG_CHECKPOINT for tablespace 1
2017-05-05 10:19:40 140037071685504 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[2201] with error Data structure corruption
2017-05-05 10:19:41 140037071685504 [Note] InnoDB: Starting shutdown...
i=================================================================
==5226==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x612000018588 in thread T0
#0 0x736750 in operator delete(void*) (/mariadb/server/build/sql/mysqld+0x736750)
#1 0x1e4833f in LatchCounter::~LatchCounter() /mariadb/server/storage/innobase/include/sync0types.h:599:4
#2 0x1e480b8 in LatchMeta<LatchCounter>::~LatchMeta() /mariadb/server/storage/innobase/include/sync0types.h:786:17
#3 0x1e35509 in sync_latch_meta_destroy() /mariadb/server/storage/innobase/sync/sync0debug.cc:1622:3
#4 0x1e35314 in sync_check_close() /mariadb/server/storage/innobase/sync/sync0debug.cc:1839:2
#5 0x1dfdc18 in innodb_shutdown() /mariadb/server/storage/innobase/srv/srv0start.cc:2888:2
#6 0x197e5e6 in innobase_init(void*) /mariadb/server/storage/innobase/handler/ha_innodb.cc:4475:3
Port of mysql changeset by Georgi Kodinov <Georgi.Kodinov@Oracle.com>:
Bug #34325 : --add-drop-trigger option for mysqldump
Implemented the --add-drop-trigger option to prepend each
CREATE TRIGGER in the dump file with DROP TRIGGER.
The option is off by default. Added a test case.
The usage of windows functions when all tables were optimized away
by min/max optimization were not supported. As result a result,
the queries that used window functions with min/max aggregation
over the whole table returned wrong result sets.
The patch fixed this problem.
The bug happened when the specification of a recursive CTE had
no recursive references at the top level of the specification.
In this case the regular processing of derived table references
of the select containing a non-recursive reference to this
recursive CTE misses handling the specification unit.
At the preparation stage any non-recursive reference to a
recursive CTE must be handled after the preparation of the
specification unit for this CTE. So we have to force this
preparation when regular handling of derived tables does not
do it.
and make sure that private ca key is not deleted at the end of
the procedure, so that we could generate additional certificates
any time without regenerating everything
- Do not throw output of exec command, if disable_result_log is set
save and dump it if exec fails. Need tha to meaningfully analyze
errors from mariabackup.
- rmdir now removes the entire tree. need that because xtrabackup tests
clean the whole directory.
- all filesystem modifying commands now require the argument to
be under MYSQLTEST_VARDIR or MYSQL_TMP_DIR.
Do not exporting mysqld entry points directly.
This is needed for mariabackup, to load encryption plugins on Windows.
All plugins are "pure" by default. To mark plugin "impure"
it should use RECOMPILE_FOR_EMBEDDED or STORAGE_ENGINE keyword.
On several buildbot machines, the test fails like this:
CURRENT_TEST: wsrep.binlog_format
mysqltest: At line 12: query 'SET binlog_format=STATEMENT' failed:
1231: Variable 'binlog_format' can't be set to the value of 'STATEMENT'
Split the test case so that a server restart is not needed.
Reduce the test cases and use a simpler mechanism for triggering
and waiting for purge.
fil_table_accessible(): Check if a table can be accessed without
enjoying MDL protection.
PROBLEM
When truncating single tablespace tables, we need to scan the entire
buffer pool to remove the pages of the table from the buffer pool.
During this scan and removal dict_sys->mutex is being held ,causing
stalls in other DDL operations.
FIX
Release the dict_sys->mutex during the scan and reacquire it after the
scan. Make sure that purge thread doesn't purge the records of the table
being truncated and background stats collection thread skips the updation
of stats for the table being truncated.
[#rb 14564 Approved by Jimmy and satya ]
Problem :
---------
Information_Schema.referential_constraints (UNIQUE_CONSTRAINT_NAME)
shows NULL for a foreign key constraint after restarting the server.
If any dml or query (select/insert/update/delete) is done on
referenced table, then the constraint name is correctly shown.
Solution :
----------
UNIQUE_CONSTRAINT_NAME column is the key name of the referenced table.
In innodb, FK reference is stored as a list of columns in referenced
table in INNODB_SYS_FOREIGN and INNODB_SYS_FOREIGN_COLS. The referenced
column must have at least one index/key with the referenced column as
prefix but the key name itself is not included in FK metadata. For this
reason, the UNIQUE_CONSTRAINT_NAME is only filled up when the
referenced table is actually loaded in innodb dictionary cache.
The information_schema view calls handler::get_foreign_key_list() on
foreign key table to read the FK metadata. The UNIQUE_CONSTRAINT_NAME
information shows NULL based on whether the referenced table is
already loaded or not.
One way to fix this issue is to load the referenced table while reading
the FK metadata information, if needed.
Reviewed-by: Sunny Bains <sunny.bains@oracle.com>
RB: 14654
Problem :
---------
This bug is filed from the base replication bug#25040331 where the
slave thread times out while INSERT operation waits on GAP lock taken
during Foreign Key validation.
The primary reason for the lock wait is because the statements are
getting replayed in different order. However, we also observed
two things ...
1. The slave thread could always use "Read Committed" isolation for
row level replication.
2. It is not necessary to have GAP locks in "READ Committed" isolation
level in innodb.
This bug is filed to address point(2) to avoid taking GAP locks during
Foreign Key validation.
Solution :
----------
Innodb is primarily designed for "Repeatable Read" and the GAP lock
behaviour is default. For "Read Committed" isolation, we have special
handling in row_search_mvcc to avoid taking the GAP lock while
scanning records.
While looking for Foreign Key, the code is following the default
behaviour taking GAP locks. The suggested fix is to avoid GAP
locking during FK validation similar to normal search operation
(row_search_mvcc) for "Read Committed" isolation level.
Reviewed-by: Sunny Bains <sunny.bains@oracle.com>
RB: 14526
Simplify the tests that are present in MySQL 5.7. Make the table
smaller while generating enough undo log. Do not unnecessarily
drop tables.
trx_purge_initiate_truncate(): Remove two crash injection points
(before and after normal redo log checkpoint), because they are
not adding any value. Clarify some messages.
trx_sys_create_rsegs(): Display the number of active undo tablespaces.
srv_undo_tablespaces_init(): When initializing the data files, do not
leave srv_undo_tablespaces_active at 0.
Do not display that number; let trx_sys_create_rsegs() display it once
the final number is known.
innodb_params_adjust(): Adjust parameters after startup.
innobase_init(): Do not allow innodb_max_undo_size to be less
than SRV_UNDO_TABLESPACE_SIZE_IN_PAGES. This avoids unnecessary
repeated truncation of undo tablespaces when using
innodb_page_size=32k or innodb_page_size=64k.
MySQL 5.7 reduced the maximum number of innodb_undo_tablespaces
from 126 to 95 when it reserved 32 persistent rollback segments
for the temporary undo logs. Since MDEV-12289 restored all 128
persistent rollback segments for persistent undo logs, the
reasonable maximum value of innodb_undo_tablespaces is 127
(not 126 or 95). This is because out of the 128 rollback segments,
the first one will always be created in the system tablespace
and the remaining ones can be created in dedicated undo tablespaces.
Analysis:
========
A foreign key constraint cannot reference a secondary index defined
on a generated virtual column. While adding new index/drop existing
column, server internally drops the internal foreign key index and
it leads to choose the virtual secondary index as foreign key index.
But innodb doesn't allow foreign key constraint reference to
secondary virtual index.
Fix:
===
Allow foreign key constraint refer to secondary index defined on
a generated virutal column.
Reviewed-by: Jimmy Yang<jimmy.yang@oracle.com>
RB: 13586
When the rows produced on the current iteration are sent to the
temporary table T of the UNION type created for CTE the rows
that were not there simultaneously are sent to the temporary
table D that contains rows for the next iteration. The test
whether a row was in T checks the return code of writing into T.
If just a HEAP table is used for T then the return code is
HA_ERR_FOUND_DUPP_KEY, but if an ARIA table is used for T then
the return code is HA_ERR_FOUND_DUPP_UNIQUE.
The implementation of select_union_recursive::send_data()
erroneously checked only for the first return code. So if an Aria
table was used for T then all rows produced by the current iteration
went to D and and in most cases D grew with each iteration.
Whether T has reached stabilization is detected by
checking whether D is empty. So as a result, the iterations were
never stopped unless a limit for them was set.
Fixed by checking for both HA_ERR_FOUND_DUPP_KEY and
HA_ERR_FOUND_DUPP_UNIQUE as return codes returned by
the function writing a row into the temporary table T.
Problem was that bpage was referenced after it was already freed
from LRU. Fixed by adding a new variable encrypted that is
passed down to buf_page_check_corrupt() and used in
buf_page_get_gen() to stop processing page read.
This patch should also address following test failures and
bugs:
MDEV-12419: IMPORT should not look up tablespace in
PageConverter::validate(). This is now removed.
MDEV-10099: encryption.innodb_onlinealter_encryption fails
sporadically in buildbot
MDEV-11420: encryption.innodb_encryption-page-compression
failed in buildbot
MDEV-11222: encryption.encrypt_and_grep failed in buildbot on P8
Removed dict_table_t::is_encrypted and dict_table_t::ibd_file_missing
and replaced these with dict_table_t::file_unreadable. Table
ibd file is missing if fil_get_space(space_id) returns NULL
and encrypted if not. Removed dict_table_t::is_corrupted field.
Ported FilSpace class from 10.2 and using that on buf_page_check_corrupt(),
buf_page_decrypt_after_read(), buf_page_encrypt_before_write(),
buf_dblwr_process(), buf_read_page(), dict_stats_save_defrag_stats().
Added test cases when enrypted page could be read while doing
redo log crash recovery. Also added test case for row compressed
blobs.
btr_cur_open_at_index_side_func(),
btr_cur_open_at_rnd_pos_func(): Avoid referencing block that is
NULL.
buf_page_get_zip(): Issue error if page read fails.
buf_page_get_gen(): Use dberr_t for error detection and
do not reference bpage after we hare freed it.
buf_mark_space_corrupt(): remove bpage from LRU also when
it is encrypted.
buf_page_check_corrupt(): @return DB_SUCCESS if page has
been read and is not corrupted,
DB_PAGE_CORRUPTED if page based on checksum check is corrupted,
DB_DECRYPTION_FAILED if page post encryption checksum matches but
after decryption normal page checksum does not match. In read
case only DB_SUCCESS is possible.
buf_page_io_complete(): use dberr_t for error handling.
buf_flush_write_block_low(),
buf_read_ahead_random(),
buf_read_page_async(),
buf_read_ahead_linear(),
buf_read_ibuf_merge_pages(),
buf_read_recv_pages(),
fil_aio_wait():
Issue error if page read fails.
btr_pcur_move_to_next_page(): Do not reference page if it is
NULL.
Introduced dict_table_t::is_readable() and dict_index_t::is_readable()
that will return true if tablespace exists and pages read from
tablespace are not corrupted or page decryption failed.
Removed buf_page_t::key_version. After page decryption the
key version is not removed from page frame. For unencrypted
pages, old key_version is removed at buf_page_encrypt_before_write()
dict_stats_update_transient_for_index(),
dict_stats_update_transient()
Do not continue if table decryption failed or table
is corrupted.
dict0stats.cc: Introduced a dict_stats_report_error function
to avoid code duplication.
fil_parse_write_crypt_data():
Check that key read from redo log entry is found from
encryption plugin and if it is not, refuse to start.
PageConverter::validate(): Removed access to fil_space_t as
tablespace is not available during import.
Fixed error code on innodb.innodb test.
Merged test cased innodb-bad-key-change5 and innodb-bad-key-shutdown
to innodb-bad-key-change2. Removed innodb-bad-key-change5 test.
Decreased unnecessary complexity on some long lasting tests.
Removed fil_inc_pending_ops(), fil_decr_pending_ops(),
fil_get_first_space(), fil_get_next_space(),
fil_get_first_space_safe(), fil_get_next_space_safe()
functions.
fil_space_verify_crypt_checksum(): Fixed bug found using ASAN
where FIL_PAGE_END_LSN_OLD_CHECKSUM field was incorrectly
accessed from row compressed tables. Fixed out of page frame
bug for row compressed tables in
fil_space_verify_crypt_checksum() found using ASAN. Incorrect
function was called for compressed table.
Added new tests for discard, rename table and drop (we should allow them
even when page decryption fails). Alter table rename is not allowed.
Added test for restart with innodb-force-recovery=1 when page read on
redo-recovery cant be decrypted. Added test for corrupted table where
both page data and FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION is corrupted.
Adjusted the test case innodb_bug14147491 so that it does not anymore
expect crash. Instead table is just mostly not usable.
fil0fil.h: fil_space_acquire_low is not visible function
and fil_space_acquire and fil_space_acquire_silent are
inline functions. FilSpace class uses fil_space_acquire_low
directly.
recv_apply_hashed_log_recs() does not return anything.
