Commit graph

460 commits

Author SHA1 Message Date
Marko Mäkelä
f693170c75 MDEV-16647 InnoDB fails to drop large temporary table on disconnect
This regression was introduced in MDEV-16515.
We would fail to drop a temporary table on client disconnect,
because trx_is_interrupted() would hold. To add insult to
injury, in MariaDB 10.1, InnoDB temporary tables are actually
persistent, so the garbage temporary tables will never be dropped.

row_drop_table_for_mysql(): If several iterations of
buf_LRU_drop_page_hash_for_tablespace() are needed,
do not interrupt dropping a temporary table even after
the transaction was marked as killed.

Server shutdown will still terminate the loop, and also DROP TABLE
of persistent tables will keep checking if the execution was aborted.
2018-08-31 13:16:31 +03:00
Marko Mäkelä
0d3972c6be Merge 10.0 into 10.1 2018-08-03 12:03:10 +03:00
Marko Mäkelä
9dfef6e29b Fix -Wclass-memaccess warnings in InnoDB,XtraDB 2018-08-03 11:53:57 +03:00
Jan Lindström
c5a8583b31 Merge tag 'mariadb-10.0.36' into 10.0-galera 2018-08-02 11:44:02 +03:00
Marko Mäkelä
87ec6a0448 Merge 10.0 into 10.1 2018-07-31 15:19:56 +03:00
Oleksandr Byelkin
865e807125 Merge branch '10.0' into 10.1 2018-07-31 11:58:29 +02:00
Marko Mäkelä
e52315a4a2 MDEV-16855 Fix fts_sync_synchronization in InnoDB
This is a backport of the following fix from MySQL 5.7.23.
Some code refactoring has been omitted, and the test case has
been adapted to MariaDB.

commit 7a689acaa65e9d602575f7aa53fe36a64a07460f
Author: Krzysztof Kapuścik <krzysztof.kapuscik@oracle.com>
Date:   Tue Mar 13 12:34:03 2018 +0100

Bug#27082268 Invalid FTS sync synchronization

The fix closes two issues:
Bug #27082268 - INNODB: FAILING ASSERTION: SYM_NODE->TABLE != NULL DURING FTS SYNC
Bug #27095935 - DEADLOCK BETWEEN FTS_DROP_INDEX AND FTS_OPTIMIZE_SYNC_TABLE

Both issues were related to a FTS cache sync being done during
operations that perfomed DDL actions on internal FTS tables
(ALTER TABLE, TRUNCATE). In some cases the FTS tables and/or
internal cache structures could get removed while still being
used to perform FTS synchronization leading to crashes. In other
the sync operations could not get finishes as it was waiting for
dict lock which was taken by thread waiting for the background
sync to be finished.

The changes done includes:
- Stopping background operations during ALTER TABLE and TRUNCATE.
- Removal of unused code in FTS.
- Cleanup of FTS sync related code to make it more readable and
easier to maintain.

RB#18262
2018-07-30 18:06:30 +03:00
Marko Mäkelä
5ed2da9587 Apply the 5.6.40 security fixes to XtraDB
We did not merge Percona XtraDB 5.6.40-84.0 yet.
The changes in it are mostly cosmetic, except for
2 bug fixes from Oracle MySQL 5.6.40, which could
be security bugs.

This was achieved by taking the applicable parts
of an earlier InnoDB commit to XtraDB:

git diff 15ec8c2f28f08517ecbffb959d756b4bdd53ab45{~,} storage/innobase|
sed -e s+/innobase/+/xtradb/+|patch -p1
2018-07-30 16:28:20 +03:00
Marko Mäkelä
d17e9a02c4 Fix InnoDB/XtraDB warnings by GCC 8.2.0 2018-07-30 14:05:24 +03:00
Marko Mäkelä
8bdd125067 MDEV-16851 On schema mismatch in IMPORT TABLESPACE, display ROW_FORMAT in clear text
This is motivated by Oracle MySQL Bug #27542720 SCHEMA MISMATCH
- TABLE FLAGS DON'T MATCH, BUT FLAGS ARE NUMBERS
but using a different approach.

row_import::match_schema(): In case of a mismatch, display the
ROW_FORMAT and optionally KEY_BLOCK_SIZE of the .cfg file.
2018-07-30 13:13:43 +03:00
Marko Mäkelä
c4eb4bcef6 MDEV-16515 InnoDB: Failing assertion: ++retries < 10000 in file
dict0dict.cc

buf_LRU_drop_page_hash_for_tablespace(): Return whether any adaptive
hash index entries existed. If yes, the caller should keep retrying to
drop the adaptive hash index.

row_import_for_mysql(), row_truncate_table_for_mysql(),
row_drop_table_for_mysql(): Ensure that the adaptive hash index was
entirely dropped for the table.
2018-06-26 11:34:51 +03:00
Jan Lindström
c69357d8d4 MDEV-15611 Due to the failure of foreign key detection, Galera slave node killed himself.
Merge following change from 10.2

revision-id: d52cff9f10aeea208a1058f7b5527e602125584c (mariadb-10.2.14-25-gd52cff9)
parent(s): bc2501453c
author: Sachin Setiya
committer: Sachin Setiya
timestamp: 2018-04-04 12:26:06 +0530
message:

MDEV-15611 Due to the failure of foreign key detection, Galera...
slave node killed himself.

Problem:- If we try to delete table with foreign key and table whom it is
referring with wsrep_slave_threads>1 then galera tries to execute both
Delete_rows_log-event in parallel, which should not happen.

Solution:- This is happening because we do not have foreign key info in
write set. Upto version 10.2.7 it used to work fine. Actually it happening
because of issue in commit 2f342c4. wsrep_must_process_fk should be used
with negation.
2018-06-15 09:04:26 +03:00
Marko Mäkelä
f5eb37129f MDEV-13103 Deal with page_compressed page corruption
fil_page_decompress(): Replaces fil_decompress_page().
Allow the caller detect errors. Remove
duplicated code. Use the "safe" instead of "fast" variants of
decompression routines.

fil_page_compress(): Replaces fil_compress_page().
The length of the input buffer always was srv_page_size (innodb_page_size).
Remove printouts, and remove the fil_space_t* parameter.

buf_tmp_buffer_t::reserved: Make private; the accessors acquire()
and release() will use atomic memory access.

buf_pool_reserve_tmp_slot(): Make static. Remove the second parameter.
Do not acquire any mutex. Remove the allocation of the buffers.

buf_tmp_reserve_crypt_buf(), buf_tmp_reserve_compression_buf():
Refactored away from buf_pool_reserve_tmp_slot().

buf_page_decrypt_after_read(): Make static, and simplify the logic.
Use the encryption buffer also for decompressing.

buf_page_io_complete(), buf_dblwr_process(): Check more failures.

fil_space_encrypt(): Simplify the debug checks.

fil_space_t::printed_compression_failure: Remove.

fil_get_compression_alg_name(): Remove.

fil_iterate(): Allocate a buffer for compression and decompression
only once, instead of allocating and freeing it for every page
that uses compression, during IMPORT TABLESPACE.

fil_node_get_space_id(), fil_page_is_index_page(),
fil_page_is_lzo_compressed(): Remove (unused code).
2018-06-14 13:46:07 +03:00
Marko Mäkelä
3627dd7f6a MDEV-16416 Crash on IMPORT TABLESPACE of a ROW_FORMAT=COMPRESSED table
fil_iterate(): Invoke fil_encrypt_buf() correctly when
a ROW_FORMAT=COMPRESSED table with a physical page size of
innodb_page_size is being imported. Also, validate the page checksum
before decryption, and reduce the scope of some variables.

AbstractCallback::operator()(): Remove the parameter 'offset'.
The check for it in FetchIndexRootPages::operator() was basically
redundant and dead code since the previous refactoring.
2018-06-07 10:34:32 +03:00
Marko Mäkelä
6aa50bad39 MDEV-16283 ALTER TABLE...DISCARD TABLESPACE still takes long on a large buffer pool
Also fixes MDEV-14727, MDEV-14491
InnoDB: Error: Waited for 5 secs for hash index ref_count (1) to drop to 0
by replacing the flawed wait logic in dict_index_remove_from_cache_low().

On DISCARD TABLESPACE, there is no need to drop the adaptive hash index.
We must drop it on IMPORT TABLESPACE, and eventually on DROP TABLE or
DROP INDEX. As long as the dict_index_t object remains in the cache
and the table remains inaccessible, the adaptive hash index entries
to orphaned pages would not do any harm. They would be dropped when
buffer pool pages are reused for something else.

btr_search_drop_page_hash_when_freed(), buf_LRU_drop_page_hash_batch():
Remove the parameter zip_size, and pass 0 to buf_page_get_gen().

buf_page_get_gen(): Ignore zip_size if mode==BUF_PEEK_IF_IN_POOL.

buf_LRU_drop_page_hash_for_tablespace(): Drop the adaptive hash index
even if the tablespace is inaccessible.

buf_LRU_drop_page_hash_for_tablespace(): New global function, to drop
the adaptive hash index.

buf_LRU_flush_or_remove_pages(), fil_delete_tablespace():
Remove the parameter drop_ahi.

dict_index_remove_from_cache_low(): Actively drop the adaptive hash index
if entries exist. This should prevent InnoDB hangs on DROP TABLE or
DROP INDEX.

row_import_for_mysql(): Drop any adaptive hash index entries for the table.

row_drop_table_for_mysql(): Drop any adaptive hash index for the table,
except if the table resides in the system tablespace. (DISCARD TABLESPACE
does not apply to the system tablespace, and we do no want to drop the
adaptive hash index for other tables than the one that is being dropped.)

row_truncate_table_for_mysql(): Drop any adaptive hash index entries for
the table, except if the table resides in the system tablespace.
2018-05-29 14:00:20 +03:00
Marko Mäkelä
35a9c90fff MDEV-14589 InnoDB should not lock a delete-marked record
When the transaction isolation level is SERIALIZABLE, or when
a locking read is performed in the REPEATABLE READ isolation level,
InnoDB must lock delete-marked records in order to prevent another
transaction from inserting something.

However, at READ UNCOMMITTED or READ COMMITTED isolation level or
when the parameter innodb_locks_unsafe_for_binlog is set, the
repeatability of the reads does not matter, and there is no need
to lock any records.

row_search_for_mysql(): Skip locks on delete-marked committed records
upfront, instead of invoking row_unlock_for_mysql() afterwards.
The unlocking never worked for secondary index records.
2018-05-29 08:54:33 +03:00
Marko Mäkelä
3b99a274a8 Merge 10.0 into 10.1 2018-05-11 17:32:20 +03:00
Sachin Agarwal
197bf0fe35 Bug #26334149 - MYSQL CRASHES WHEN FULL TEXT INDEXES IBD FILES ARE ORPHANED DUE TO RENAME TABLE
Problem:
When FTS index is added into a table which doesn't have 'FTS_DOC_ID'
column, Innodb rebuilds table to add column 'FTS_DOC_ID'. when this FTS
index is dropped from this table. Innodb doesn't not rebuild table to
remove 'FTS_DOC_ID' column and deletes FTS index auxiliary tables.
But it doesn't delete FTS common auxiliary tables.
Later when the database having this table is renamed, FTS auxiliary
tables are not renamed because table's flags2 (dict_table_t.flags2)
has been resetted for DICT_TF2_FTS flag during FTS index drop operation.
Now when we drop old database, it leads to an assert.

Fix:
During renaming of FTS auxiliary tables, ORed a condition to check if
table has DICT_TF2_FTS_HAS_DOC_ID flag set.

RB: 18769
Reviewed by : Jimmy.Yang@oracle.com
2018-05-11 17:26:40 +03:00
Thirunarayanan Balathandayuthapani
9c03ba8f0d Bug #27041445 SERVER ABORTS IF FTS_DOC_ID EXCEEDS FTS_DOC_ID_MAX_STEP
Problem:
=======
Multiple insert statement in table contains FULLTEXT KEY and a
FTS_DOC_ID column aborts the server if the FTS_DOC_ID exceeds
FTS_DOC_ID_MAX_STEP.

Solution:
========
Remove the exception for first committed insert statement.

Reviewed-by: Jimmy Yang<jimmy.yang@oracle.com>
RB: 18023
2018-05-11 15:52:45 +03:00
Marko Mäkelä
580a8061a7 Remove a redundant condition added by the 5.6.40 merge
When Oracle fixed MDEV-13899 in their own way, they moved the
condition to the only caller of PageConverter::update_records().
Thus, the merge of 5.6.40 into MariaDB added a redundant condition.

PageConverter::update_records(): Move the page_is_leaf() condition
to the only caller, PageConverter::update_index_page().
2018-05-11 14:02:13 +03:00
Marko Mäkelä
d06ca5bbf6 Merge 10.0 into 10.1 2018-05-09 15:58:04 +03:00
Marko Mäkelä
4f42f0d1ea MDEV-16119 InnoDB lock->index refers to a freed object after failed ADD INDEX
The problem is hard to repeat, and I failed to create a deterministic
test case. Online index creation creates stubs for to-be-created indexes.
If index creation fails, we could remove these stubs while locks exist
in the indexes. (This would require that the index creation was completed,
and a concurrent DML operation acquired a lock on a record in the
uncommitted index. If a duplicate key error occurs in an uncommitted
index, the error will be reported for the CREATE UNIQUE INDEX, not for
the DML operation that tried to insert the duplicate.)

dict_table_try_drop_aborted(), row_merge_drop_indexes(): If transactional
locks exist on the table, keep the table->indexes intact.
2018-05-09 15:56:26 +03:00
Jan Lindström
e1ffb66449 Merge tag 'mariadb-10.0.35' into 10.0-galera 2018-05-07 17:20:39 +03:00
Sergei Golubchik
9989c26bc9 Merge branch '10.0' into 10.1 2018-05-05 14:01:59 +02:00
Mayank Prasad
f47eac2882 Bug #25928471: ONLINE ALTER AND CONCURRENT DELETE ON TABLE WITH MANY TEXT COLUMNS CAUSES CRASH
Issue:
------
Prefix for externally stored columns were being stored in online_log when a
table is altered and alter causes table to be rebuilt. Space in online_log is
limited and if length of prefix of externally stored columns is very big, then
it is being written to online log without making sure if it fits. This leads to
memory corruption.

Fix:
----
After fix for Bug#16544143, there is no need to store prefixes of externally
stored columnd in online_log. Thus remove the code which stores column prefixes
for externally stored columns. Also, before writing anything on online_log,
make sure it fits to available memory to avoid memory corruption.

Read RB page for more details.

Reviewed-by: Annamalai Gurusami <annamalai.gurusami@oracle.com>
RB: 18239
2018-05-03 11:41:21 +03:00
Sergei Golubchik
6b84fdb2f3 5.6.39-83.1 2018-04-24 16:17:43 +02:00
Jan Lindström
a5001a2ad7 Merge tag 'mariadb-5.5.60' into 5.5-galera 2018-04-24 13:34:57 +03:00
Jan Lindström
3855943fe0 MDEV-12632: Source and destination overlap in memcpy, encryption.innodb-discard-import-change fails in buildbot with valgrind
Use block->page.offset for checking page number.
2018-04-13 09:25:52 +03:00
Jan Lindström
0ae13b51d7 MDEV-12632: Source and destination overlap in memcpy, encryption.innodb-discard-import-change fails in buildbot with valgrind
Problem was that if tablespace was encrypted we try to copy
also page 0 from read buffer to write buffer that are in
that case the same memory area.

fil_iterate
	When tablespace is encrypted or compressed its
        first page (i.e. page 0) is not encrypted or
	compressed and there is no need to copy buffer.
2018-04-12 11:00:48 +03:00
Marko Mäkelä
8eff803a1b Revert "MDEV-14705: Do not rollback on InnoDB shutdown"
This reverts commit 76ec37f522.

This behaviour change will be done separately in:
MDEV-15832 With innodb_fast_shutdown=3, skip the rollback
of connected transactions
2018-04-10 08:55:20 +03:00
Marko Mäkelä
76ec37f522 MDEV-14705: Do not rollback on InnoDB shutdown
row_undo_step(): If fast shutdown has been requested, abort the
rollback of any non-DDL transactions. Starting with MDEV-12323,
we aborted the rollback of recovered transactions. These
transactions would be rolled back on subsequent server startup.

trx_roll_report_progress(): Renamed from trx_roll_must_shutdown(),
now that the shutdown check has been moved to the only caller.
2018-04-06 09:58:14 +03:00
Marko Mäkelä
e0a0fe7d81 MDEV-12396 IMPORT TABLESPACE: Do not retry partial reads
fil_iterate(), fil_tablespace_iterate(): Replace os_file_read()
with os_file_read_no_error_handling().

os_file_read_func(), os_file_read_no_error_handling_func():
Do not retry partial reads. There used to be an infinite amount
of retries. Because InnoDB extends both data and log files upfront,
partial reads should be impossible during normal operation.
2018-03-20 15:31:39 +02:00
Marko Mäkelä
a80af35a85 MDEV-12396 IMPORT cleanup: ROW_FORMAT=COMPRESSED
Initialize block.page.zip only once.

PageConverter::update(): Initialize m_page_zip_ptr
as late as possible.
(We should really remove it at some point.)

PageConverter::operator(): Refer to block->page.zip instead of
m_page_zip_ptr.

AbstractCallback::get_frame(): Define static. Refer
to block->page.zip.data directly.

fil_iterate(): Refer to block->page.zip.data directly.

fil_tablespace_iterate(): Initialize block.page.zip.data as soon
as possible.
2018-03-20 15:31:39 +02:00
Marko Mäkelä
eaa7bfb59f MDEV-12396 IMPORT TABLESPACE: Simplify validation
fil_iterate(): Validate the pages directly.

import_page_status_t, PageConverter::validate(): Remove.

AbstractCallback::filename(): New accessor.

AbstractCallback::is_interrupted(): Replaces periodic_check().

PageConverter::trigger_corruption(): Remove.
2018-03-20 15:31:39 +02:00
Marko Mäkelä
6247c64c2a MDEV-12396 IMPORT TABLESPACE cleanup
Reduce unnecessary inter-module calls for IMPORT TABLESPACE.
Move some IMPORT-related code from fil0fil.cc to row0import.cc.

PageCallback: Remove. Make AbstractCallback the base class.

PageConverter: Define some member functions inline.
2018-03-20 15:31:39 +02:00
Thirunarayanan Balathandayuthapani
eee73ddfbb MDEV-12255 innodb_prefix_index_cluster_optimization hits debug build
assert on UTF-8 columns

Problem:
=======
(1) Multi-byte character cases are not considered during prefix index
cluster optimization check. It leads to fetch of improper results during
read operation.
(2) Strict assert in row_sel_field_store_in_mysql_format_func and it asserts
for prefix index record to mysql conversion.

Solution:
========
(1) Consider the case of multi-byte character during prefix index
cluster optimization check.
(2) Relax the assert in row_sel_field_store_in_mysql_format_func to allow
prefix index record to mysql format conversion.

The patch is taken from
1eee538087
2018-03-20 17:53:33 +05:30
Marko Mäkelä
ca40330d1d Fix a deadlock in thd_report_wait_for()
Unlike commit a54abf0175 claimed,
the caller of THD::awake() may actually hold the InnoDB lock_sys->mutex.
That commit introduced a deadlock of threads in the replication slave
when running the test rpl.rpl_parallel_optimistic_nobinlog.

lock_trx_handle_wait(): Expect the callers to acquire and release
lock_sys->mutex and trx->mutex.

innobase_kill_query(): Restore the logic for conditionally acquiring
and releasing the mutexes. THD::awake() can be called from inside
InnoDB while holding one or both mutexes, via thd_report_wait_for() and
via wsrep_innobase_kill_one_trx().
2018-03-16 08:23:56 +02:00
Alexey Botchkov
b88542681b MDEV-14611 ALTER TABLE EXCHANGE PARTITION does not work properly when used with DATA DIRECTORY.
When table is renamed, the InnoDB's dictionary cache didn't
        change the ib_table->data_dir_path accordingly.
        Now it's set to NULL.
2018-02-10 22:17:49 +04:00
Marko Mäkelä
871f2a6ee2 Merge 10.0 into 10.1 2018-02-08 13:29:08 +02:00
Marko Mäkelä
5421e3aee7 MDEV-15249 Crash in MVCC read after IMPORT TABLESPACE
PageConverter::adjust_cluster_record(): Instead of writing
the invalid value DB_ROLL_PTR=0, write a value that indicates
a fresh insert, that is, prevents the DB_ROLL_PTR from being
dereferenced in any circumstances.

It can be argued that IMPORT TABLESPACE should actually
update the dict_index_t::trx_id to prevent older transactions
from accessing the table, similar to what I did on table
rebuild in MySQL 5.6.6 in
03f81a55f2
2018-02-08 12:59:32 +02:00
Sergei Golubchik
d4df7bc9b1 Merge branch 'github/10.0' into 10.1 2018-02-02 10:09:44 +01:00
Jan Lindström
c7e5feb259 Merge tag 'mariadb-10.0.34' into 10.0-galera
Conflicts:
	storage/innobase/lock/lock0lock.cc
	storage/xtradb/lock/lock0lock.cc
	storage/xtradb/lock/lock0wait.cc
	support-files/mysql.server.sh
2018-02-01 14:09:48 +02:00
Marko Mäkelä
706ed8552d Revert "MDEV-6928: Add trx pointer to struct mtr_t"
This reverts commit 3486135bb5.

The commit comment ended in the words: "This is needed later."
Apparently the "later" never arrived.
2018-01-29 11:05:17 +02:00
Vicențiu Ciorbaru
a82bb5d316 Merge branch 'merge-xtradb-5.6' into 10.0 2018-01-24 20:20:11 +02:00
Vicențiu Ciorbaru
d833bb65d5 Merge remote-tracking branch '5.5' into 10.0 2018-01-24 12:29:31 +02:00
Marko Mäkelä
70a9b12de9 Silence -Wimplicit-fallthrough 2018-01-23 18:08:55 +02:00
Vicențiu Ciorbaru
cd33250d2a 5.6.38-83.0 2018-01-23 18:04:34 +02:00
Marko Mäkelä
906ce0962d MDEV-7049 MySQL#74585 - InnoDB: Failing assertion: *mbmaxlen < 5 in file ha_innodb.cc line 1904
InnoDB limited the maximum number of bytes per character to 4.
But, the filename character set that was introduced in MySQL 5.1
uses up to 5 bytes per character.

To allow InnoDB tables to be created with wider characters, let
us split the mbminmaxlen fields into mbminlen, mbmaxlen, and increase
the limit to 7 bytes per character. This will increase the payload size
of dtype_t and dict_col_t by one bit. The storage size will be unchanged
(54 bits and 77 bits will use the same number of bytes as the
previous sizes 53 and 76 bits).
2018-01-22 11:18:10 +02:00
Jan Lindström
ff979674e6 Merge tag 'mariadb-5.5.59' into 5.5-galera 2018-01-19 19:22:00 +02:00
Marko Mäkelä
5e87f49a99 Make row_mysql_table_id_reassign() static 2018-01-18 09:29:49 +02:00
Sergei Golubchik
85a5e58d71 5.5.58-38.10 2018-01-17 17:28:33 +01:00
Marko Mäkelä
016caa3d20 Merge 10.0 into 10.1 2018-01-02 21:57:22 +02:00
Marko Mäkelä
51e4650ed0 Merge 5.5 into 10.0 2018-01-02 21:52:46 +02:00
Marko Mäkelä
d384ead0f0 MDEV-14799 After UPDATE of indexed columns, old values will not be purged from secondary indexes
This is a regression caused by MDEV-14051 'Undo log record is too big.'

Purge in the secondary index is wrongly skipped in
row_purge_upd_exist_or_extern() because node->row only does not contain all
indexed columns.

trx_undo_rec_get_partial_row(): Add the parameter for node->update
so that the updated columns will be copied from the initial part
of the undo log record.
2018-01-02 19:11:10 +02:00
Marko Mäkelä
9d7c0882fa Merge 10.0 into 10.1 2017-12-21 17:25:51 +02:00
Marko Mäkelä
0202e47274 MDEV-12827 Assertion failure when reporting duplicate key error in online table rebuild
row_log_table_apply_insert_low(), row_log_table_apply_update():
When reporting the error_key_num, only count the clustered index
if it corresponds to a key in the SQL layer.

The assertion failure was probably introduced by the (incomplete)
MySQL 5.6.28 bug fix
Bug #21364096 THE BOGUS DUPLICATE KEY ERROR IN ONLINE DDL
WITH INCORRECT KEY NAME
which we are improving.

Side note: the fix was incorrectly merged to MySQL 5.7.10;
incorrect key names will continue to be reported in MySQL 5.7.
2017-12-21 17:19:13 +02:00
Marko Mäkelä
09c5bbf471 Merge 10.0 into 10.1 2017-12-18 20:05:50 +02:00
Marko Mäkelä
40088bfc7e MDEV-13407 innodb.drop_table_background failed in buildbot with "Tablespace for table exists"
The InnoDB background DROP TABLE queue is something that we should
really remove, but are unable to until we remove dict_operation_lock
so that DDL and DML operations can be combined in a single transaction.

Because the queue is not persistent, it is not crash-safe. In stable
versions of MariaDB, we can only try harder to drop all enqueued
tables before server shutdown.

row_mysql_drop_t::table_id: Replaces table_name.

row_drop_tables_for_mysql_in_background():
Do not remove the entry from the list as long as the table exists.
In this way, the table should eventually be dropped.
2017-12-18 19:51:44 +02:00
Marko Mäkelä
46305b006b Merge 10.0 into 10.1 2017-12-13 19:12:16 +02:00
Marko Mäkelä
b1977a39de MDEV-12323 Rollback progress log messages during crash recovery are intermixed with unrelated log messages
trx_roll_must_shutdown(): During the rollback of recovered transactions,
report progress and check if the rollback should be interrupted because
of a pending shutdown.

trx_roll_max_undo_no, trx_roll_progress_printed_pct: Remove, along with
the messages that were interleaved with other messages.
2017-12-13 18:56:22 +02:00
Marko Mäkelä
b46fa627ca MDEV-12352 InnoDB shutdown should not be blocked by a large transaction rollback
row_undo_step(), trx_rollback_active(): Abort the rollback of a
recovered ordinary transaction if fast shutdown has been initiated.

trx_rollback_resurrected(): Convert an aborted-rollback transaction
into a fake XA PREPARE transaction, so that fast shutdown can proceed.
2017-12-13 18:02:09 +02:00
Jan Lindström
d7349e204b MDEV-9663: InnoDB assertion failure: *cursor->index->name == TEMP_INDEX_PREFIX
MariaDB adjustments to test case innodb-replace-debug and
add missing instrumentation to row0ins.cc.

MariaDB 10.1 does not seem to be affected.
2017-11-16 14:05:49 +02:00
Jan Lindström
d8ccc61f76 MDEV-9663: InnoDB assertion failure: *cursor->index->name == TEMP_INDEX_PREFIX
Add missing instrumentation to row0ins.cc.
2017-11-16 14:03:02 +02:00
Sergei Golubchik
2a4e4335c4 Merge branch 'github/10.0-galera' into 10.1 2017-11-10 01:38:03 +01:00
sjaakola
f4f2e8fa2a MW-402 cascading FK issues
* created tests focusing in multi-master conflicts during cascading foreign key
  processing
* in row0upd.cc, calling wsrep_row_ups_check_foreign_constraints only when
  running in cluster
* in row0ins.cc fixed regression from MW-369, which caused crash with MW-402.test
2017-11-08 14:15:41 +02:00
Marko Mäkelä
51b4366bfb MDEV-13328 ALTER TABLE…DISCARD TABLESPACE takes a lot of time
With a big buffer pool that contains many data pages,
DISCARD TABLESPACE took a long time, because it would scan the
entire buffer pool to remove any pages that belong to the tablespace.
With a large buffer pool, this would take a lot of time, especially
when the table-to-discard is empty.

The minimum amount of work that DISCARD TABLESPACE must do is to
remove the pages of the to-be-discarded table from the
buf_pool->flush_list because any writes to the data file must be
prevented before the file is deleted.

If DISCARD TABLESPACE does not evict the pages from the buffer pool,
then IMPORT TABLESPACE must do it, because we must prevent pre-DISCARD,
not-yet-evicted pages from being mistaken for pages of the imported
tablespace.

It would not be a useful fix to simply move the buffer pool scan to
the IMPORT TABLESPACE step. What we can do is to actively evict those
pages that could be mistaken for imported pages. In this way, when
importing a small table into a big buffer pool, the import should
still run relatively fast.

Import is bypassing the buffer pool when reading pages for the
adjustment phase. In the adjustment phase, if a page exists in
the buffer pool, we could replace it with the page from the imported
file. Unfortunately I did not get this to work properly, so instead
we will simply evict any matching page from the buffer pool.

buf_page_get_gen(): Implement BUF_EVICT_IF_IN_POOL, a new mode
where the requested page will be evicted if it is found. There
must be no unwritten changes for the page.

buf_remove_t: Remove. Instead, use trx!=NULL to signify that a write
to file is desired, and use a separate parameter bool drop_ahi.

buf_LRU_flush_or_remove_pages(), fil_delete_tablespace():
Replace buf_remove_t.

buf_LRU_remove_pages(), buf_LRU_remove_all_pages(): Remove.

PageConverter::m_mtr: A dummy mini-transaction buffer

PageConverter::PageConverter(): Complete the member initialization list.

PageConverter::operator()(): Evict any 'shadow' pages from the
buffer pool so that pre-existing (garbage) pages cannot be mistaken
for pages that exist in the being-imported file.

row_discard_tablespace(): Remove a bogus comment that seems to
refer to IMPORT TABLESPACE, not DISCARD TABLESPACE.
2017-11-06 18:08:33 +02:00
Sachin Setiya
3cecb1bab3 Merge tag 'mariadb-10.0.33' into bb-10.0-galera 2017-11-03 12:34:05 +05:30
Marko Mäkelä
38e12db478 Merge 10.0 into 10.1 2017-10-26 13:36:38 +03:00
Marko Mäkelä
b933a8c354 MDEV-12569 InnoDB suggests filing bugs at MySQL bug tracker
Replace all references in InnoDB and XtraDB error log messages
to bugs.mysql.com with references to https://jira.mariadb.org/.

The original merge
commit 4274d0bf57
was accidentally reverted by the subsequent merge
commit 3b35d745c3
2017-10-26 13:29:28 +03:00
Vicențiu Ciorbaru
3b35d745c3 Merge branch 'merge-innodb-5.6' into 10.0 2017-10-26 12:46:47 +03:00
Marko Mäkelä
4274d0bf57 Merge 5.5 into 10.0 2017-10-26 11:13:07 +03:00
Marko Mäkelä
cfb3361748 MDEV-12569 InnoDB suggests filing bugs at MySQL bug tracker
Replace all references in InnoDB and XtraDB error log messages
to bugs.mysql.com with references to https://jira.mariadb.org/.
2017-10-26 11:02:19 +03:00
Sergei Golubchik
9d2e2d7533 Merge branch '10.0' into 10.1 2017-10-22 13:03:41 +02:00
Jan Lindström
8da6b4ef52 Merge tag 'mariadb-5.5.58' into 5.5-galera 2017-10-19 09:06:17 +03:00
Sergei Golubchik
da4503e956 Merge branch '5.5' into 10.0 2017-10-18 15:14:39 +02:00
Sergei Golubchik
d76f5774fe MDEV-13459 Warnings, when compiling with gcc-7.x
mostly caused by -Wimplicit-fallthrough
2017-10-17 07:37:39 +02:00
Jan Lindström
fc9ff69578 MDEV-13838: Wrong result after altering a partitioned table
Reverted incorrect changes done on MDEV-7367 and MDEV-9469. Fixes properly
also related bugs:

MDEV-13668: InnoDB unnecessarily rebuilds table when renaming a column and adding index
MDEV-9469: 'Incorrect key file' on ALTER TABLE
MDEV-9548: Alter table (renaming and adding index) fails with "Incorrect key file for table"
MDEV-10535: ALTER TABLE causes standalone/wsrep cluster crash
MDEV-13640: ALTER TABLE CHANGE and ADD INDEX on auto_increment column fails with "Incorrect key file for table..."

Root cause for all these bugs is the fact that MariaDB .frm file
can contain virtual columns but InnoDB dictionary does not and
previous fixes were incorrect or unnecessarily forced table
rebuilt. In index creation key_part->fieldnr can be bigger than
number of columns in InnoDB data dictionary. We need to skip not
stored fields when calculating correct column number for InnoDB
data dictionary.

dict_table_get_col_name_for_mysql
        Remove

innobase_match_index_columns
        Revert incorrect change done on MDEV-7367

innobase_need_rebuild
        Remove unnecessary rebuild force when column is renamed.

innobase_create_index_field_def
        Calculate InnoDB column number correctly and remove
        unnecessary column name set.

innobase_create_index_def, innobase_create_key_defs
        Remove unneeded fields parameter. Revert unneeded memset.

prepare_inplace_alter_table_dict
        Remove unneeded col_names parameter

index_field_t
        Remove unneeded col_name member.

row_merge_create_index
        Remove unneeded col_names parameter and resolution.

Effected tests:
         innodb-alter-table : Add test case for MDEV-13668
         innodb-alter : Remove MDEV-13668, MDEV-9469 FIXMEs
                        and restore original tests
         innodb-wl5980-alter : Remove MDEV-13668,  MDEV-9469 FIXMEs
                        and restore original tests
2017-10-10 17:03:40 +03:00
Marko Mäkelä
172cc70bf8 MDEV-13446 fts_create_doc_id() unnecessarily allocates 8 bytes for every inserted row
fts_create_doc_id(): Remove.

row_mysql_convert_row_to_innobase(): Implement the logic of
fts_create_doc_id(). Reuse a buffer for the hidden FTS_DOC_ID.

row_get_prebuilt_insert_row(): Allocate a buffer for the hidden
FTS_DOC_ID at the end of prebuilt->ins_upd_rec_buff.
2017-10-09 12:18:12 +03:00
Jan Lindström
4d01dd79a1 MDEV-12634: Uninitialised ROW_MERGE_RESERVE_SIZE bytes written to temporary file
After review cleanup.
2017-09-28 12:38:51 +03:00
Sachin Setiya
0627929f62 MDEV-13787 Crash in persistent stats wsrep_on (thd=0x0)
Problem:- This crash happens because of thd = NULL , and while checking
for wsrep_on , we no longer check for thd != NULL (MDEV-7955). So this
problem is regression of MDEV-7955. However this patch not only solves
this regression , It solves all regression caused by MDEV-7955 patch.

To get all possible cases when thd can be null , assert(thd)/
assert(trx->mysql_thd) is place just before all wsrep_on and innodb test
suite is run. And the assert which caused failure are removed with a physical
check for thd != NULL. Rest assert are removed. Hopefully this method will
remove all current/potential regression of MDEV-7955.
2017-09-27 10:15:08 +05:30
Marko Mäkelä
84be33abe0 Merge 10.0 into 10.1 2017-09-25 09:50:24 +03:00
Marko Mäkelä
78b63425a3 MDEV-13899 IMPORT TABLESPACE may corrupt ROW_FORMAT=REDUNDANT tables
The ALTER TABLE…IMPORT TABLESPACE adjustment code that was introduced by
WL#5522 in MySQL 5.6 is incorrectly invoking rec_get_status() on a
ROW_FORMAT=REDUNDANT record to determine if a record is a leaf page record.
The function rec_get_status(rec) is only to be called on
ROW_FORMAT=COMPACT, DYNAMIC or COMPRESSED records.
2017-09-24 10:11:16 +03:00
Vicențiu Ciorbaru
ec6042bda0 Merge branch '10.0' into 10.1 2017-09-19 12:06:50 +03:00
Vicențiu Ciorbaru
b337a06829 Revert d9bc5e03d788b958ce8c76e157239953db60adb2 from Oracle (in XtraDB) 2017-09-19 11:53:59 +03:00
Vicențiu Ciorbaru
dab6f56098 Revert Bug #25175249 ASSERTION: (TEMPL->IS_VIRTUAL && !FIELD) || (FIELD && FIELD->PREFIX_LEN ? FIELD
We don't want this in innodb/xtradb
2017-09-19 11:46:38 +03:00
Vicențiu Ciorbaru
cbd62feee5 Merge branch 'merge-xtradb-5.6' into 10.0 2017-09-19 11:35:09 +03:00
Vicențiu Ciorbaru
b4606367d7 5.6.37-82.2 2017-09-19 00:44:27 +03:00
Marko Mäkelä
836d4e74d9 Write proper tests for MDEV-12634: Uninitialised ROW_MERGE_RESERVE_SIZE bytes
Introduce innodb_encrypt_log.combinations and prove that
the encryption and decryption take place during both
online ADD INDEX (WL#5266) and online table-rebuilding ALTER (WL#6625).
2017-09-16 21:15:38 +03:00
Jan Lindström
fa2701c6f7 MDEV-12634: Uninitialised ROW_MERGE_RESERVE_SIZE bytes written to tem…
…porary file

Fixed by removing writing key version to start of every block that
was encrypted. Instead we will use single key version from log_sys
crypt info.

After this MDEV also blocks writen to row log are encrypted and blocks
read from row log aren decrypted if encryption is configured for the
table.

innodb_status_variables[], struct srv_stats_t
	Added status variables for merge block and row log block
	encryption and decryption amounts.

Removed ROW_MERGE_RESERVE_SIZE define.

row_merge_fts_doc_tokenize
	Remove ROW_MERGE_RESERVE_SIZE

row_log_t
	Add index, crypt_tail, crypt_head to be used in case of
	encryption.

row_log_online_op, row_log_table_close_func
	Before writing a block encrypt it if encryption is enabled

row_log_table_apply_ops, row_log_apply_ops
	After reading a block decrypt it if encryption is enabled

row_log_allocate
	Allocate temporary buffers crypt_head and crypt_tail
	if needed.

row_log_free
	Free temporary buffers crypt_head and crypt_tail if they
	exist.

row_merge_encrypt_buf, row_merge_decrypt_buf
	Removed.

row_merge_buf_create, row_merge_buf_write
	Remove ROW_MERGE_RESERVE_SIZE

row_merge_build_indexes
	Allocate temporary buffer used in decryption and encryption
	if needed.

log_tmp_blocks_crypt, log_tmp_block_encrypt, log_temp_block_decrypt
	New functions used in block encryption and decryption

log_tmp_is_encrypted
	New function to check is encryption enabled.

Added test case innodb-rowlog to force creating a row log and
verify that operations are done using introduced status
variables.
2017-09-14 09:23:20 +03:00
Marko Mäkelä
cd694d76ce Merge 10.0 into 10.1 2017-09-06 15:32:56 +03:00
Marko Mäkelä
6b45355e6b MDEV-13103 Assertion `flags & BUF_PAGE_PRINT_NO_CRASH' failed in buf_page_print
buf_page_print(): Remove the parameter 'flags',
and when a server abort is intended, perform that in the caller.

In this way, page corruption reports due to different reasons
can be distinguished better.

This is non-functional code refactoring that does not fix any
page corruption issues. The change is only made to avoid falsely
grouping together unrelated causes of page corruption.
2017-09-06 14:01:15 +03:00
Marko Mäkelä
7f99381288 Fix compiler warnings
metadata_lock_info_duration[]: Remove the unused variable.

Add some comments /* fall through */ to silence -Wimplicit-fallthrough
2017-09-01 15:40:49 +03:00
Marko Mäkelä
4c91fd4cd6 Galera after-merge fixes
wsrep_drop_table_query(): Remove the definition of this ununsed function.

row_upd_sec_index_entry(), row_upd_clust_rec_by_insert():
Evaluate the simplest conditions first. The merge could have slightly
hurt performance by causing extra calls to wsrep_on().
2017-08-30 12:29:47 +03:00
Jan Lindström
01209de763 Merge remote-tracking branch 'origin/bb-10.1-jplindst' into 10.1 2017-08-29 20:30:18 +03:00
Marko Mäkelä
11352d52cd Merge 10.0 into 10.1 2017-08-28 15:05:46 +03:00
Marko Mäkelä
582545a384 MDEV-13637 InnoDB change buffer housekeeping can cause redo log overrun and possibly deadlocks
The function ibuf_remove_free_page() may be called while the caller
is holding several mutexes or rw-locks. Because of this, this
housekeeping loop may cause performance glitches for operations that
involve tables that are stored in the InnoDB system tablespace.
Also deadlocks might be possible.

The worst impact of all is that due to the mutexes being held, calls to
log_free_check() had to be skipped during this housekeeping.
This means that the cyclic InnoDB redo log may be overwritten.
If the system crashes during this, it would be unable to recover.

The entry point to the problematic code is ibuf_free_excess_pages().
It would make sense to call it before acquiring any mutexes or rw-locks,
in any 'pessimistic' operation that involves the system tablespace.

fseg_create_general(), fseg_alloc_free_page_general(): Do not call
ibuf_free_excess_pages() while potentially holding some latches.

ibuf_remove_free_page(): Do call log_free_check(), like every operation
that is about to generate redo log should do.

ibuf_free_excess_pages(): Remove some assertions that are replaced
by stricter assertions in the log_free_check() that is now called by
ibuf_remove_free_page().

row_ins_sec_index_entry(), row_undo_ins_remove_sec_low(),
row_undo_mod_del_mark_or_remove_sec_low(),
row_undo_mod_del_unmark_sec_and_undo_update(): Call
ibuf_free_excess_pages() if the operation may involve allocating pages
and change buffering in the system tablespace.
2017-08-25 14:01:51 +03:00
Jan Lindström
c23efc7d50 Merge remote-tracking branch 'origin/10.0-galera' into 10.1 2017-08-21 13:35:00 +03:00
sjaakola
318af3f3b8 MW-369 FK fixes
Skipping wsrep extra FK checking for applier and replayer threads
2017-08-14 14:35:44 +03:00
sjaakola
396770fb67 Refs: MW-369 * changed parent row key type to S(hared), when FK child table is being updated or deleted 2017-08-14 13:40:37 +03:00
sjaakola
41fac0a54c Refs: MW-369 * changed insert for a FK child table to take exclusive lock on FK parent table 2017-08-14 13:15:46 +03:00