Commit graph

19391 commits

Author SHA1 Message Date
Sergey Vojtovich
b3346c2f41 Restore LF_BACKOFF
Moved InnoDB UT_RELAX_CPU() to server. Restored cross-platform LF_BACKOFF
implementation basing on UT_RELAX_CPU().
2017-12-08 13:44:45 +02:00
Marko Mäkelä
07e9ff1fe1 MDEV-14378 In ALGORITHM=INPLACE, use a common name for the intermediate tables or partitions
Allow DROP TABLE `#mysql50##sql-...._.` to drop tables that were
being rebuilt by ALGORITHM=INPLACE

NOTE: If the server is killed after the table-rebuilding ALGORITHM=INPLACE
commits inside InnoDB but before the .frm file has been replaced, then
the recovery will involve something else than DROP TABLE.

NOTE: If the server is killed in a true inplace ALTER TABLE commits
inside InnoDB but before the .frm file has been replaced, then we
are really out of luck. To properly handle that situation, we would
need a transactional mysql.ddl_fixup table that directs recovery to
rename or remove files.

prepare_inplace_alter_table_dict(): Use the altered_table->s->table_name
for generating the new_table_name.

table_name_t::part_suffix: The start of the partition name suffix.

table_name_t::dbend(): Return the end of the schema name.

table_name_t::dblen(): Return the length of the schema name, in bytes.

table_name_t::basename(): Return the name without the schema name.

table_name_t::part(): Return the partition name, or NULL if none.

row_drop_table_for_mysql(): Assert for #sql, not #sql-ib.
2017-12-08 10:07:51 +02:00
Marko Mäkelä
0a02c1a667 Merge 10.2 into bb-10.2-ext 2017-12-08 10:02:28 +02:00
Marko Mäkelä
dfafe15abb MDEV-14606 Assertion failure on IMPORT TABLESPACE
fseg_alloc_free_page_low(): Remove a bogus and redundant assertion about
fil_space_t::purpose. The debug function fsp_space_modify_check()
is asserting something similar, but more accurately.
2017-12-08 09:53:11 +02:00
Jan Lindström
da3a3a68df MDEV-12837: WSREP: BF lock wait long
Problem was a merge error from MySQL wsrep i.e. Galera.

wsrep_on_check
	New check function. Galera can't be enabled
	if innodb-lock-schedule-algorithm=VATS.

innobase_kill_query
	In Galera async kill we could own lock mutex.

innobase_init
	If Variance-Aware-Transaction-Sheduling Algorithm (VATS) is
	used on Galera we fall back to First-Come-First-Served (FCFS)
	with notice to user.

Changed innodb-lock-schedule-algorithm as read-only parameter
as it was designed to be.

lock_reset_lock_and_trx_wait
	Use ib::hex() to print out transaction ID.

lock_rec_other_has_expl_req,
lock_rec_other_has_conflicting,
RecLock::add_to_waitq
lock_rec_lock_slow
lock_table_other_has_incompatible
lock_rec_insert_check_and_lock
lock_prdt_other_has_conflicting

	Change pointer to conflicting lock to normal pointer as this
	pointer contents could be changed later.

RecLock::create
	Conclicting lock pointer is moved to last parameter with
	default value NULL. This conflicting transaction could
	be selected as victim in Galera if requesting transaction
	is BF (brute force) transaction. In this case contents
	of conflicting lock pointer will be changed. Use ib::hex() to print
	transaction ids.
2017-12-07 13:08:41 +02:00
Marko Mäkelä
bce4065129 Merge 10.2 into bb-10.2-ext 2017-12-07 08:18:43 +02:00
Marko Mäkelä
51c73a431f Merge 10.1 into 10.2 2017-12-07 08:17:50 +02:00
Marko Mäkelä
447931c6ab Post-fix for MDEV-14587
dict_stats_process_entry_from_defrag_pool(): Release the mutex
2017-12-07 08:14:49 +02:00
Marko Mäkelä
ce07676502 Merge 10.2 into bb-10.2-ext 2017-12-06 19:34:03 +02:00
Marko Mäkelä
77fb7ccba4 Follow-up fix to MDEV-13201 Assertion srv_undo_sources || ... failed on shutdown during DDL operation
Introduce the debug flag trx_t::persistent_stats to suppress the
assertion for the updates of persistent statistics during fast
shutdown.

dict_stats_exec_sql(): Do execute the statement even though shutdown
has been initiated.
2017-12-06 18:52:28 +02:00
Marko Mäkelä
7dc6066dea MDEV-14511 Use fewer transactions for updating InnoDB persistent statistics
dict_stats_exec_sql(): Expect the caller to always provide a transaction.
Remove some redundant assertions. The caller must hold dict_sys->mutex,
but holding dict_operation_lock is only necessary for accessing
data dictionary tables, which we are not accessing.

dict_stats_save_index_stat(): Acquire dict_sys->mutex
for invoking dict_stats_exec_sql().

dict_stats_save(), dict_stats_update_for_index(), dict_stats_update(),
dict_stats_drop_index(), dict_stats_delete_from_table_stats(),
dict_stats_delete_from_index_stats(), dict_stats_drop_table(),
dict_stats_rename_in_table_stats(), dict_stats_rename_in_index_stats(),
dict_stats_rename_table(): Use a single caller-provided
transaction that is started and committed or rolled back by the caller.

dict_stats_process_entry_from_recalc_pool(): Let the caller provide
a transaction object.

ha_innobase::open(): Pass a transaction to dict_stats_init().

ha_innobase::create(), ha_innobase::discard_or_import_tablespace():
Pass a transaction to dict_stats_update().

ha_innobase::rename_table(): Pass a transaction to
dict_stats_rename_table(). We do not use the same transaction
as the one that updated the data dictionary tables, because
we already released the dict_operation_lock. (FIXME: there is
a race condition; a lock wait on SYS_* tables could occur
in another DDL transaction until the data dictionary transaction
is committed.)

ha_innobase::info_low(): Pass a transaction to dict_stats_update()
when calculating persistent statistics.

alter_stats_norebuild(), alter_stats_rebuild(): Update the
persistent statistics as well. In this way, a single transaction
will be used for updating the statistics of a whole table, even
for partitioned tables.

ha_innobase::commit_inplace_alter_table(): Drop statistics for
all partitions when adding or dropping virtual columns, so that
the statistics will be recalculated on the next handler::open().
This is a refactored version of Oracle Bug#22469660 fix.

RecLock::add_to_waitq(), lock_table_enqueue_waiting():
Do not allow a lock wait to occur for updating statistics
in a data dictionary transaction, such as DROP TABLE. Instead,
return the previously unused error code DB_QUE_THR_SUSPENDED.

row_merge_lock_table(), row_mysql_lock_table(): Remove dead code
for handling DB_QUE_THR_SUSPENDED.

row_drop_table_for_mysql(), row_truncate_table_for_mysql():
Drop the statistics as part of the data dictionary transaction.
After TRUNCATE TABLE, the statistics will be recalculated on
subsequent ha_innobase::open(), similar to how the logic after
the above-mentioned Oracle Bug#22469660 fix in
ha_innobase::commit_inplace_alter_table() works.

btr_defragment_thread(): Use a single transaction object for
updating defragmentation statistics.

dict_stats_save_defrag_stats(), dict_stats_save_defrag_stats(),
dict_stats_process_entry_from_defrag_pool(),
dict_defrag_process_entries_from_defrag_pool(),
dict_stats_save_defrag_summary(), dict_stats_save_defrag_stats():
Add a parameter for the transaction.

dict_stats_empty_table(): Make public. This will be called by
row_truncate_table_for_mysql() after dropping persistent statistics,
to clear the memory-based statistics as well.
2017-12-06 18:52:28 +02:00
Sergei Petrunia
2c1e4d4d7a MDEV-14563: Wrong query plan for query with no PK
Part #2: Don't use the new code for the clustered PK, it is handled
in the special way right above.
2017-12-06 12:35:17 +03:00
Sergei Petrunia
a6254e5e7d MDEV-14563: Wrong query plan for query with no PK
TABLE_SHARE::init_from_binary_frm_image() calls handler_file->index_flags()
before it has set TABLE_SHARE::primary_key (it is 0 while it should be
MAX_KEY in my example).
This causes MyRocks to report wrong index flags (it thinks it's a PK while
it is not), which causes invalid query plans later on.

Do the only thing that seems feasible: adjust field->part_of key to have
correct value in ha_rocksdb::open.
2017-12-06 12:35:17 +03:00
Sergei Petrunia
c3803914c5 MDEV-14433: RocksDB may show empty or incorrect output with rocksdb_strict_collation_check=off
Part#1: Set field->part_of_key correctly for PK fields.
2017-12-06 12:35:17 +03:00
Marko Mäkelä
e9bc0f75ef MDEV-5834 cleanup: Inline two tiny functions 2017-12-06 10:32:24 +02:00
Marko Mäkelä
1d526f31fb Merge 10.1 into 10.2 2017-12-05 14:23:57 +02:00
Marko Mäkelä
63cbb98275 MDEV-14587 dict_stats_process_entry_from_defrag_pool() fails to call dict_table_close() when index==NULL
dict_stats_process_entry_from_defrag_pool(): Simplify the logic,
and always call dict_table_close() when dict_table_open() returned
a non-NULL handle.
2017-12-05 13:25:09 +02:00
Marko Mäkelä
d1ab89037a MDEV-13670/MDEV-14550 Error log flood : "InnoDB: page_cleaner: 1000ms intended loop took N ms. The settings might not be optimal."
Silence the error log output that was introduced in MySQL 5.7
(MariaDB 10.2.2) if log_warnings=2 or less.

We should still figure out what these messages really indicate
and how to solve the problems.

pc_sleep_if_needed(): Add a parameter for the current time,
so that there will be fewer successive calls to ut_time_ms()
with no I/O between them.

buf_flush_page_cleaner_coordinator(): Exit the first loop
whenever shutdown has been requested. At the start of the loop,
call ut_time_ms() only once. Do not display the message if
log_warnings=2 or less.
2017-12-05 12:58:09 +02:00
Marko Mäkelä
8be7548085 Follow-up to MDEV-12698: Adjust some comments
The function dict_stats_update_if_needed() replaced
row_update_statistics_if_needed(). Adjust the comments accordingly.
2017-12-04 13:43:02 +02:00
Marko Mäkelä
bd8fd3b7c3 Remove references to UNIV_SYNC_DEBUG which was merged with UNIV_DEBUG 2017-12-04 11:48:12 +02:00
Marko Mäkelä
24c9785a67 Merge 10.2 into bb-10.2-ext 2017-12-01 15:35:16 +02:00
Marko Mäkelä
8d24bef640 MDEV-14080 InnoDB shutdown sometimes hangs
srv_purge_wakeup(): If thd_destructor_proxy has initiated the first
step of shutdown, ensure that all purge threads terminate.

logs_empty_and_mark_files_at_shutdown(): Add a debug assertion.
(The purge threads should have been shut down already before this step.)
2017-12-01 15:02:04 +02:00
Vladislav Vaintroub
f59a1826f8 MDEV-14536 : during backup, retry read of log blocks, if there is
(possibly intermittent) checksum mismatch.
2017-11-30 16:30:40 +00:00
Alexander Barkov
5b697c5a23 Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext 2017-11-29 12:06:48 +04:00
Vladislav Vaintroub
9cefffdab1 Merge remote-tracking branch 'origin/10.1' into 10.2
# Conflicts:
#	storage/innobase/include/os0file.h
2017-11-24 17:30:26 +00:00
Vladislav Vaintroub
40756c9151 Fix Windows build with -DPLUGIN_PERFSCHEMA=NO 2017-11-24 16:55:20 +00:00
Marko Mäkelä
f1cc6e3874 Merge 10.1 into 10.2 2017-11-24 17:17:16 +02:00
Sergei Golubchik
0055e1a57f Merge branch 'connect/10.2' into 10.2 2017-11-22 17:38:30 +01:00
David Carlier
f86413ecc1 Fix the build on OpenBSD (#488)
* rocksdb fails without timer_delete() - only use it when it exists
2017-11-22 12:52:12 +01:00
wlad
b6d72ed44d MDEV-14283 : Fix Solaris 10 build.
- introduce system check for posix_memalign (not available on Solaris 10)
- Disable dtrace probes, to fix weird link errors in mariabackup
2017-11-21 21:14:06 +01:00
Sergei Golubchik
14c2a9a52e don't git submodule update from rocksdb/CMakeLists.txt (#492)
we now have cmake/submodules.cmake that updates all submodules
2017-11-21 20:03:57 +01:00
Sergei Golubchik
a963cb95db MDEV-14268 Executable C source code files? 2017-11-21 20:03:57 +01:00
Sergei Golubchik
7f1900705b Merge branch '10.1' into 10.2 2017-11-21 19:47:46 +01:00
Marko Mäkelä
366bb162fe Merge 10.2 into bb-10.2-ext 2017-11-21 18:59:30 +02:00
Marko Mäkelä
9405fdeb9e MDEV-13201 Assertion srv_undo_sources || ... failed on shutdown during DDL operation
dict_stats_exec_sql(): Refuse the operation if shutdown has been
initiated.

The real fix would be to update the persistent statistics as part
of the data dictionary transactions. To do this, we should move the
storage of InnoDB persistent statistics to the InnoDB data files,
and maybe also remove the InnoDB data dictionary.
2017-11-21 17:56:31 +02:00
Marko Mäkelä
f233c9778e Adjust the MySQL 5.7 tests for MariaDB 10.2 2017-11-20 16:05:41 +02:00
Marko Mäkelä
ce64a65f27 MDEV-14310 Possible corruption by table-rebuilding or index-creating ALTER TABLE…ALGORITHM=INPLACE
Also, MDEV-14317 When ALTER TABLE is aborted, do not write garbage pages to data files

As pointed out by Shaohua Wang, the merge of MDEV-13328 from
MariaDB 10.1 (based on MySQL 5.6) to 10.2 (based on 5.7)
was performed incorrectly.

Let us always pass a non-NULL FlushObserver* when writing
to data files is desired.

FlushObserver::is_partial_flush(): Check if this is a bulk-load
(partial flush of the tablespace).

FlushObserver::is_interrupted(): Check for interrupt status.

buf_LRU_flush_or_remove_pages(): Instead of trx_t*, take
FlushObserver* as a parameter.

buf_flush_or_remove_pages(): Remove the parameters flush, trx.
If observer!=NULL, write out the data pages. Use the new predicate
observer->is_partial() to distinguish a partial tablespace flush
(after bulk-loading) from a full tablespace flush (export).
Return a bool (whether all pages were removed from the flush_list).

buf_flush_dirty_pages(): Remove the parameter trx.
2017-11-20 13:26:56 +02:00
Alexander Barkov
4a8039b04e Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext 2017-11-20 11:12:08 +04:00
David Carlier
685db2c112 Fix the build on OpenBSD (#488)
* cast pthread_t for printf
* don't use RTLD_NOLOAD
* tokudb fails without F_NOCACHE and O_DIRECT - ditto
2017-11-17 13:55:14 +08:00
Sergei Golubchik
4666f01534 mroonga after-merge test fixes
"sed -r" fails on labrador. Don't use sed, use perl.
2017-11-17 13:55:14 +08:00
Vladislav Vaintroub
0d1e805aeb Fix rocksdb tests on Windows 2017-11-16 22:11:08 +00:00
Vladislav Vaintroub
689168be12 MDEV-13852 - redefine WinWriteableFile such as IsSyncThreadSafe()
is set to true, as it should.

Copy and modify original io_win.h header file to a different location
(as we cannot patch anything in submodule). Make sure modified header is
used.
2017-11-16 18:57:18 +00:00
Jan Lindström
d7349e204b MDEV-9663: InnoDB assertion failure: *cursor->index->name == TEMP_INDEX_PREFIX
MariaDB adjustments to test case innodb-replace-debug and
add missing instrumentation to row0ins.cc.

MariaDB 10.1 does not seem to be affected.
2017-11-16 14:05:49 +02:00
Jan Lindström
0c4d11e819 MDEV-13206: INSERT ON DUPLICATE KEY UPDATE foreign key fail
This is caused by following change:

commit 95d29c99f01882ffcc2259f62b3163f9b0e80c75
Author: Marko Mäkelä <marko.makela@oracle.com>
Date:   Tue Nov 27 11:12:13 2012 +0200

    Bug#15920445 INNODB REPORTS ER_DUP_KEY BEFORE CREATE UNIQUE INDEX COMPLETED

    There is a phase during online secondary index creation where the index has
    been internally completed inside InnoDB, but does not 'officially' exist yet.
    We used to report ER_DUP_KEY in these situations, like this:

    ERROR 23000: Can't write; duplicate key in table 't1'

    What we should do is to let the 'offending' operation complete, but report an
    error to the
    ALTER TABLE t1 ADD UNIQUE KEY (c2):

    ERROR HY000: Index c2 is corrupted
    (This misleading error message should be fixed separately:
    Bug#15920713 CREATE UNIQUE INDEX REPORTS ER_INDEX_CORRUPT INSTEAD OF DUPLICATE)

    row_ins_sec_index_entry_low(): flag the index corrupted instead of
    reporting a duplicate, in case the index has not been published yet.

    rb:1614 approved by Jimmy Yang

Problem is that after we have found duplicate key on primary key
we continue to get necessary gap locks in secondary indexes to
block concurrent transactions from inserting the searched records.
However, search from unique index used in foreign key constraint
could return DB_NO_REFERENCED_ROW if INSERT .. ON DUPLICATE KEY UPDATE
does not contain value for foreign key column. In this case
we should return the original DB_DUPLICATE_KEY error instead
of DB_NO_REFERENCED_ROW.

Consider as a example following:

create table child(a int not null primary key,
b int not null,
c int,
unique key (b),
foreign key (b) references
parent (id)) engine=innodb;

insert into child values (1,1,2);

insert into child(a) values (1) on duplicate key update c = 3;

Now primary key value 1 naturally causes duplicate key error that will be
stored on node->duplicate. If there was no duplicate key error, we should
return the actual no referenced row error. As value for column b used in
both unique key and foreign key is not provided, server uses 0 as a
search value. This is naturally, not found leading to DB_NO_REFERENCED_ROW.
But, we should update the row with primay key value 1 anyway as
requested by on duplicate key update clause.
2017-11-16 11:05:24 +02:00
Jun Su
2401d14e6b Support CRC32 SSE2 implementation under Windows 2017-11-16 09:59:18 +01:00
Marko Mäkelä
5e7435c4b0 Correct a merge error (remove a bogus #ifdef) 2017-11-15 08:53:31 +02:00
Vladislav Vaintroub
f48d56c459 MDEV-14192 Apply marko's patch 2017-11-13 14:58:18 +00:00
Sergei Golubchik
8b18a44fa7 mroonga after-merge CMakeLists.txt fixes
1. remove erroneously committed *.orig
2. fix LZ4 detection on Mac OS X and FreeBSD. Cannot do

   pkg_check_modules(LIBLZ4 liblz4)
   find_library(LIBLZ4_LIBS ... )

because find_library(X) does not do anything if X is defined (documented),
and pkg_check_modules(Y) sets Y_LIBS to "" (undocumented!)
2017-11-13 13:11:53 +08:00
Marko Mäkelä
b2dd5232d4 InnoDB: Remove ut_vsnprintf() and the use of my_vsnprintf(); use vsnprintf() 2017-11-13 04:32:56 +02:00
Marko Mäkelä
c19ef508b8 InnoDB: Remove ut_snprintf() and the use of my_snprintf(); use snprintf() 2017-11-13 02:11:48 +02:00