Commit graph

20519 commits

Author SHA1 Message Date
Sergey Vojtovich
7ad9b3588f OS X warnings fixes 2018-05-20 15:55:56 +04:00
Sergei Golubchik
ff1ec6e958 cleanup: create_temp_file()
simplify. move common code inside, specify common flags inside,
rewrite dead code (`if (mode & O_TEMPORARY)` on Linux, where
`O_TEMPORARY` is always 0) to actually do something.
2018-05-19 22:18:46 +02:00
Elena Stepanova
e4e0aea644 Follow-up for be6ae0bb6b - fix test results
Duplicate warnings have been removed
2018-05-19 14:23:06 +03:00
Thirunarayanan Balathandayuthapani
f38d8c1701 MDEV-16201 CREATE TABLE creates extra transaction
InnoDB does not allow FOREIGN KEY constraints to exist for TEMPORARY TABLE.
InnoDB introduced a dedicated tablespace for temporary tables, and actually
stopped creating persistent metadata and data for temporary tables.

row_table_add_foreign_constraints(): Do not create a persistent
transaction.

dict_create_foreign_constraints_low(): Add the persistent transaction to
the update the foreign key relation in dictionary.

dict_create_foreign_constraints_low(): Remove a duplicated check for
partitioned tables.
2018-05-17 22:03:53 +05:30
Sergei Golubchik
15425767e8 Merge remote-tracking branch 'github/10.3' into 10.3 2018-05-17 15:48:50 +02:00
Aleksey Midenkov
60319afff7 MDEV-15969 System versioning and FEDERATED don't work well together: DML and discovery fail
* fixed discovery
* DELETE fix
* federated.test

Closes 
2018-05-17 15:33:42 +02:00
Sergei Golubchik
dfd6702a29 MDEV-16157 federated corrupts timestamps
do "set time_zone='+00:00'" on remote server
after connecting.

temporarily reset local time zone to my_tz_OFFSET0
whenever Field::val_str() or Field::store()
methods are used.
2018-05-17 15:33:39 +02:00
Marko Mäkelä
b2f56085d4 MDEV-16045: Fix a heap allocation
Since commit baa5a43d8c
changed the allocation of log_t::files::file_header_bufs[],
a memory overrun could occur when running with a non-default
setting of innodb_log_files_in_group.

log_t::files::create(): Correctly allocate n_files*2048 + 512
bytes of memory, to be aligned at a 512-byte boundary.
2018-05-17 15:21:32 +03:00
Marko Mäkelä
4c7608aeb1 Merge 10.2 into 10.3 2018-05-17 08:42:53 +03:00
Marko Mäkelä
a4e7800701 MDEV-13779 InnoDB fails to shut down purge workers, causing hang
srv_purge_coordinator_thread(): Wait for all purge worker threads
to actually exit. An analysis of a core dump of a hung 10.3 server
revealed that one srv_worker_thread did not exit, even though the
purge coordinator had exited. This caused kill_server_thread and
mysqld_main to wait indefinitely. The main InnoDB shutdown was
never called, because unireg_end() was never called.
2018-05-16 16:35:33 +03:00
Marko Mäkelä
66360506f0 MDEV-16168: Resurrect the record MLOG_UNDO_INIT
While the record type MLOG_UNDO_INIT feels redundant, it does save
redo log volume, and subtle changes to redo log volume appear to
make a big difference for the log_sys.mutex contention.

trx_undo_parse_page_init(): Allow type==0, which is what we write
since MDEV-12288. Parse type in a simpler way; it always was written
as a single byte.

trx_undo_page_init(): Write a MLOG_UNDO_INIT record.
2018-05-16 10:18:18 +03:00
Marko Mäkelä
cd15e764a8 MDEV-16159 Use atomic memory access for purge_sys
Thanks to Sergey Vojtovich for feedback and many ideas.

purge_state_t: Remove. The states are replaced with
purge_sys_t::enabled() and purge_sys_t::paused() as follows:
PURGE_STATE_INIT, PURGE_STATE_EXIT, PURGE_STATE_DISABLED: !enabled().
PURGE_STATE_RUN, PURGE_STATE_STOP: paused() distinguishes these.

purge_sys_t::m_paused: Renamed from purge_sys_t::n_stop.
Protected by atomic memory access only, not purge_sys_t::latch.

purge_sys_t::m_enabled: An atomically updated Boolean that
replaces purge_sys_t::state.

purge_sys_t:🏃 Remove, because it duplicates
srv_sys.n_threads_active[SRV_PURGE].

purge_sys_t::running(): Accessor for srv_sys.n_threads_active[SRV_PURGE].

purge_sys_t::stop(): Renamed from trx_purge_stop().

purge_sys_t::resume(): Renamed from trx_purge_run().
Do not acquire latch; solely rely on atomics.

purge_sys_t::is_initialised(), purge_sys_t::m_initialised: Remove.

purge_sys_t::create(), purge_sys_t::close(): Instead of invoking
is_initialised(), check whether event is NULL.

purge_sys_t::event: Move before latch, so that fields that are
protected by latch can reside on the same cache line with latch.

srv_start_wait_for_purge_to_start(): Merge to the only caller srv_start().
2018-05-15 23:01:18 +03:00
Marko Mäkelä
442a6e6b25 MDEV-16172 Remove InnoDB 5.7 version number from MariaDB 10.3 onwards
Because the InnoDB implementation in MariaDB has diverged from MySQL,
it is not meaningful to report a MySQL version number for InnoDB
any more. Some examples include:

MariaDB 10.1 (which is based on MySQL 5.6) included encryption and
variable-size page compression before MySQL 5.7 introduced them.
MariaDB 10.2 (based on MySQL 5.7) introduced persistent AUTO_INCREMENT
(MDEV-6076) in a GA release before MySQL 8.0.
MariaDB 10.3 (based on MySQL 5.7) introduced instant ADD COLUMN
(MDEV-11369) before MySQL.

All of these features use a different implementation and file format.
Also, some features were never merged from MySQL 5.7, and thus MariaDB
is not affected by related bugs. Examples include CREATE TABLESPACE
and the reimplementation of the partitioning engine.
2018-05-15 23:01:18 +03:00
Marko Mäkelä
27f12c5d5b MDEV-16143 Assertion failure in rec_offs_get_n_alloc() due to row_trx_id_offset()
row_trx_id_offset(): Add the missing rec_offs_init() call.
This was broken by commit 97e51d24cb.
2018-05-15 23:01:18 +03:00
Marko Mäkelä
93b69825ad MDEV-16169 InnoDB: Failing assertion: !space->referenced()
Before invoking fil_space_t::acquire(), check space->is_stopping()
to ensure that the tablespace actually is accessible. This fixes
a regression introduced by MDEV-15983.

fil_space_next(): Remove some duplicated code for prev_space==NULL,
and check is_stopping() also on the first tablespace.

i_s_tablespaces_encryption_fill_table(),
i_s_tablespaces_scrubbing_fill_table(): Check is_stopping().
2018-05-15 23:01:18 +03:00
Marko Mäkelä
2b812abd1a Correct a comment 2018-05-15 23:01:18 +03:00
Sergei Petrunia
06f539dab5 Revert the following patch brought in by "Merge pull request from shinnok/10.3-macfixes"
It caused compile failures:

commit 6620fbd62a
Author: Teodor Mircea Ionita <teodor@mariadb.org>
Date:   Thu May 10 12:23:35 2018 +0300

    MDEV-15778: On macOS pthread_t is opaque, requires explicit cast

    On macOS pthread id is a pointer to struct _opaque_pthread_t type,
    requires explicit cast to ulint which in turn is size_t;
    Was failing with Clang 9.1.0 Debug build on macOS 10.13.4:

    sync0policy.h:53:4: error: cannot initialize a member subobject of type 'ulint'
    (aka 'unsigned long') with an rvalue of type 'os_thread_id_t' (aka '_opaque_pthread_t *')
    m_thread_id(os_thread_id_t(ULINT_UNDEFINED))
                            ^           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    sync0policy.h:79:4: error: cannot initialize a parameter of type 'int64' (aka 'long long') with
    an rvalue of type 'os_thread_id_t' (aka '_opaque_pthread_t *')
    my_atomic_storelint(&m_thread_id, os_thread_get_curr_id());
2018-05-15 17:17:09 +03:00
Sergei Golubchik
0e296947db add missing test result
followup for 21bcfeb996
2018-05-15 15:13:33 +02:00
Monty
e7fc8cd683 MDEV-654 Assertion `share->now_transactional' failed in flush_log_for_bitmap on concurrent workload with Aria tables
I was able to repeat the problem with old version of randgen

Reason for crash:

- It's not safe to change share->now_transactional if there are changed
bitmaps in the pagecache as flushing these can cause redo-entries and
the bitmap flush code checks that share->now_transactional is set.

Fixed by flushing bitmaps in _ma_tmp_disable_logging_for_table() before
we set share->now_transactional to 0
2018-05-15 13:51:03 +03:00
Sergei Petrunia
21bcfeb996 MDEV-16155: UPDATE on RocksDB table with unique constraint does not work
RocksDB now supports "iterator bounds" which are min and max keys
that an iterator is interested in.

Iterator initialization function doesn't copy the keys, though, it keeps
pointers to them.
So if the buffer space for the keys is used for another iterator (the one
for checking for UNIUQE constraint violation in ha_rocksdb::ha_update_row)
then one can get incorrect query result.

Fixed by using a separate buffer for iterator bounds in the unique constraint
violation check.
2018-05-15 12:34:10 +03:00
Sergei Petrunia
a0048378f9
Merge pull request from shinnok/10.3-macfixes
MDEV-15778: 10.3 macOS build fixes
2018-05-15 12:20:16 +03:00
Marko Mäkelä
681e8ca35e MDEV-16142: Update the InnoDB version number to 5.7.22 2018-05-15 08:18:44 +03:00
Sergei Petrunia
0d033b6d34 Down-scale rocksdb.bulk_load* tests by 2x. This should fix MDEV-13904 2018-05-14 23:54:43 +03:00
Marko Mäkelä
2b24b04220 Bug 27122803 - BACKPORT FIX FOR BUG 25899959 TO MYSQL-5.7
Merge a test case and a code change from MySQL 5.7.22.
There was no commit message, but a test case was included.
d3ec326bcd

There is no Bug 25899959 mentioned in the MySQL 8.0.11 history.
Based on the number, it should have been filed before August 2017.
Maybe it was initially fixed in a not-yet-public MySQL 9.0 branch?

The code change differs from MySQL 5.7, because the mbminmaxlen
were split in MariaDB in MDEV-7049.
2018-05-14 23:29:13 +03:00
Sergei Petrunia
4d2a36e8bc Post-merge fix for rocksdb.add_index_inplace_sstfilewriter
MariaDB has a scaled-down version of the test so we need to set
@@rocksdb_max_row_locks lower to trigger the desired error

(didn't catch this on test BB run because this test is marked as "big")
2018-05-14 20:25:02 +03:00
Sergei Petrunia
279184a04d MDEV-14562: rocksdb.bloomfilter failed in buildbot
Bloom filter is only used when reading the data from disk. If the data
happens to be still in the memtable, bloomfilter wont be used.

Stabilize the testcase by making sure the data is on disk before we
read it.
2018-05-14 19:51:19 +03:00
Marko Mäkelä
fb5d579462 MDEV-13987 Hang in FLUSH TABLES...FOR EXPORT
trx_purge_stop(): Release purge_sys->latch before attempting to
wake up the purge threads, so that they can actually wake up.
This is a regression of commit a13a636c74
which attempted to fix MDEV-11802 by ensuring that srv_purge_wakeup()
will actually wait until all purge threads wake up.
Due to the purge_sys->latch, the threads might never wake up,
because some purge threads could end up waiting for purge_sys->latch
in trx_undo_prev_version_build() while holding dict_operation_lock
in shared mode. This in turn would block any DDL operations, the
InnoDB master thread, and InnoDB shutdown.
2018-05-14 18:19:20 +03:00
Sergei Petrunia
16a8a241c2 MDEV-14375: rocksdb.bulk_load_unsorted fails
Down-scale the test by a factor of 2.
2018-05-14 17:57:33 +03:00
Sergei Petrunia
7e7592ade5 MDEV-16154: Server crashes in in myrocks::ha_rocksdb::load_auto_incr_value_from_index
Backport the fix from the upstream and add our testcase.
Backported cset:
commit 997a979bf5e2f75ab88781d9d3fd22dddc1fc21f
Author: Manuel Ung <mung@fb.com>
Date:   Thu Feb 15 08:38:12 2018 -0800

    Fix crashes in autoincrement code paths

    Summary:
    There are two issues related to autoincrement that can lead to crashes:

    1. The max value for double/float type for autoincrement was not implemented in MyRocks, and can lead to assertions. The fix is to add them in.
    2. If we try to set auto_increment via alter table on a table without an auto_increment column defined, we segfault because there is no index from which to read the last value. The fix is to perform a check to see if autoincrement exists before reading from index (similar to code ha_rocksdb::open).

    Fixes https://github.com/facebook/mysql-5.6/issues/792
    Closes https://github.com/facebook/mysql-5.6/pull/794

    Differential Revision: D6995096

    Pulled By: lth

    fbshipit-source-id: 1130ce1
2018-05-14 15:59:51 +03:00
Marko Mäkelä
15419a5583 Merge 10.2 into 10.3 2018-05-12 22:14:59 +03:00
Marko Mäkelä
5e84ea9634 MDEV-12266: Remove dict_table_is_discarded()
The predicate dict_table_is_discarded() checks whether
ALTER TABLE…DISCARD TABLESPACE has been executed.

Replace most occurrences of dict_table_is_discarded() with
checks of dict_table_t::space. A few checks for the flag
DICT_TF2_DISCARDED are necessary; write them inline.

Because !is_readable() implies !space, some checks for
dict_table_is_discarded() were redundant.
2018-05-12 22:12:12 +03:00
Marko Mäkelä
c57e9835ff Replace dict_col_is_virtual(col) with col->is_virtual() 2018-05-12 22:12:12 +03:00
Marko Mäkelä
ba43914ec4 Replace dict_table_is_temporary(table) with table->is_temporary() 2018-05-12 22:12:12 +03:00
Sergei Golubchik
7bf39888f7 CONNECT: remove an outdated declaration 2018-05-12 10:16:45 +02:00
Marko Mäkelä
77867c147b dict_create_index_tree_in_mem(): Remove dead code
In InnoDB, CREATE TEMPORARY TABLE does not allow FULLTEXT INDEX.
Replace a condition with a debug assertion, and add a test.
2018-05-12 10:28:54 +03:00
Sachin Agarwal
0da98472b7 Bug CRASH IN BUF_BLOCK_FROM_AHI WHEN LARGE PAGES AND AHI ARE ENABLED
Problem:

Fix for Bug  (#Rb9581) introduced a conditional debug execute
'buf_pool_resize_chunk_null', which causes new chunks memory for 2nd
buffer pool instance is freed.

Buffer pool resize function removes all old chunks entry from
'buf_chunk_map_reg' and add new chunks entry into it. But when
'buf_pool_resize_chunk_null' is set true, 2nd buffer pool
instance's chunk entries are not added into 'buf_chunk_map_reg'.
When purge thread tries to access that buffer chunk, it leads to
debug assertion.

Fix:

Added old chunk entries into 'buf_chunk_map_reg' for 2nd buffer pool
instance when 'buf_pool_resize_chunk_null' debug condition is set to true.

Reviewed by: Jimmy <Jimmy.Yang@oracle.com>
RB: 18664
2018-05-11 19:10:32 +03:00
Aakanksha Verma
4c7ea34ef6 FOLLOW-UP FIX FOR BUG#27141613
PROBLEM

Issue found during ntest run is a regression of  Bug . The
issue is basically when index is being freed due to an error during its
creation,when the index isn't added to dictionary cache its field
columns are not set, the derefrencing of null col pointer during the
clean of index from the virtual column's leads to a crash.

NOTE: Also test i_innodb.virtual_debug was failing on 32k page size and
above for the newly added scenario. Fixed that.

FIX

Added a check that if only the index is cached , the virtual index
freeing from the virtual cols index list is performed.

Reviewed by: Satya Bodapati<satya.bodapati@oracle.com>
RB: 18670
2018-05-11 19:10:32 +03:00
Aakanksha Verma
279f992b85 Bug ASSERTION: TRX0REC.CC:319:COL->IS_VIRTUAL() / CRASH IN TRX_UNDO_READ_V_COLS
PROBLEM
=======
When add of virtual index fails with DB_TOO_BIG_RECORD , the virtual
index being freed isn't removed from the list of indexes a virtual
column(which is part of the index). This while the undo log is read
could fetch a wrong value during rollback and cause the assertion
reported in the bug particularly.

FIX
===
Added a function that is called when the virtual index being freed would
allow the index be removed from the index list of virtual column which
was a field of that index.

Reviwed By: Jimmy Yang<Jimmy.Yang@oracle.com>
RB: 18528
2018-05-11 19:10:32 +03:00
Aditya A
280879ebd9 Bug MYSQL CRASH DOING SYNC INDEX ] [FATAL] INNODB: SEMAPHORE WAIT HAS LASTED > 600
PROBLEM
-------

Whenever an fts table is created it registers itself in a queue which
is operated by a background thread whose job is to optimize the
fts tables in background. Additionally we place these fts tables in
non-LRU list so that they cannot be evicted from cache. But in the
scenario when a node is brought up which is already having fts
tables ,we first try to load the fts tables in dictionary ,but we skip
the part where it is added in background queue and in non-LRU list because
the background thread is not yet created,so these tables are loaded
but they can be evicted from the cache. Now coming to the deadlock scenario

1. A Server background thread is trying to evict a table from the cache
    because the cache is full,so it scans the LRU list for the tables it can
    evict.It finds the fts table (because of the reason explained above)
    can be evicted and it takes the dict_sys->mutex (this is a system wide mutex)
    submits a request to  the background thread to remove this table from queue
    and waits it to be completed.

2.  In the mean time fts_optimize_thread() is processing another job
    in the queue and needs dict_sys->mutex for a small amount of time,
    but it cannot get it because it is blocked by the first background thread.

So Thread 1 is waiting for its job to be completed by Thread 2,whereas Thread 2
is waiting for dict_sys->mutex held by thread 1 ,causing the deadlock.

FIX
2018-05-11 19:10:32 +03:00
Sachin Agarwal
b7e333f98a Bug INNODB COMPLAINS OF SYNTAX ERROR, BUT DOES NOT SAY WHICH OPTION
Problem:

when incorrect value is assigned to innodb_data_file_path or
innodb_temp_data_file_path parameter, Innodb returns error and logs error
message in mysqlds.err file but there is no information in error message about
the parameter which causes Innodb initialization is failed.

Fix:

Added error message with parameter name and value, which causes Innodb
initialization is failed.

Reviewed by: Jimmy <Jimmy.Yang@oracle.com>
RB: 18206
2018-05-11 19:10:32 +03:00
Sachin Agarwal
3d10966b7d Bug FK ON TABLE WITH GENERATED COLS: ASSERTION POS < N_DEF
Problem:

During ALTER, when filling stored column info, wrong column number is used.
This is because we ignored virtual column when iterating over columns in
table and lead to debug assertion.

Fix:

In InnoDB table cache object, vcols are on stored on one list, stored and
normal columns are stored in another list.
When looking for stored column, ignore the virtual columns to get the right
column number of stored column.

Reviewed by: Thiru <thirunarayanan.balathandayuth@oracle.com>,
             Satya <satya.bodapati@oracle.com>
RB: 17939
2018-05-11 18:35:03 +03:00
Marko Mäkelä
82f0dc35aa Merge 10.1 into 10.2 2018-05-11 18:33:58 +03:00
Marko Mäkelä
c407ee0976 MDEV-16145 Crash in ALTER TABLE…AUTO_INCREMENT=1 after DISCARD TABLESPACE
This is the MariaDB equivalent of fixing the MySQL 5.7 regression
Bug  ALTER TABLE AUTO_INCREMENT TRIES TO READ
INDEX FROM DISCARDED TABLESPACE

Oracle did not publish a test case, but it is easy to guess
based on the commit message. The MariaDB code is different
due to MDEV-6076 implementing persistent AUTO_INCREMENT.

commit_set_autoinc(): Report ER_TABLESPACE_DISCARDED if the
tablespace is missing.

prepare_inplace_alter_table_dict(): Avoid accessing a discarded
tablespace. (This avoids generating warnings in fil_space_acquire().)
2018-05-11 18:30:14 +03:00
Marko Mäkelä
64f4576be4 MDEV-15697 post-fix: Remove an unused variable 2018-05-11 18:09:05 +03:00
Marko Mäkelä
3b99a274a8 Merge 10.0 into 10.1 2018-05-11 17:32:20 +03:00
Sachin Agarwal
197bf0fe35 Bug - MYSQL CRASHES WHEN FULL TEXT INDEXES IBD FILES ARE ORPHANED DUE TO RENAME TABLE
Problem:
When FTS index is added into a table which doesn't have 'FTS_DOC_ID'
column, Innodb rebuilds table to add column 'FTS_DOC_ID'. when this FTS
index is dropped from this table. Innodb doesn't not rebuild table to
remove 'FTS_DOC_ID' column and deletes FTS index auxiliary tables.
But it doesn't delete FTS common auxiliary tables.
Later when the database having this table is renamed, FTS auxiliary
tables are not renamed because table's flags2 (dict_table_t.flags2)
has been resetted for DICT_TF2_FTS flag during FTS index drop operation.
Now when we drop old database, it leads to an assert.

Fix:
During renaming of FTS auxiliary tables, ORed a condition to check if
table has DICT_TF2_FTS_HAS_DOC_ID flag set.

RB: 18769
Reviewed by : Jimmy.Yang@oracle.com
2018-05-11 17:26:40 +03:00
Thirunarayanan Balathandayuthapani
9c03ba8f0d Bug SERVER ABORTS IF FTS_DOC_ID EXCEEDS FTS_DOC_ID_MAX_STEP
Problem:
=======
Multiple insert statement in table contains FULLTEXT KEY and a
FTS_DOC_ID column aborts the server if the FTS_DOC_ID exceeds
FTS_DOC_ID_MAX_STEP.

Solution:
========
Remove the exception for first committed insert statement.

Reviewed-by: Jimmy Yang<jimmy.yang@oracle.com>
RB: 18023
2018-05-11 15:52:45 +03:00
Sergei Golubchik
c9717dc019 Merge branch '10.2' into 10.3 2018-05-11 13:15:10 +02:00
Marko Mäkelä
580a8061a7 Remove a redundant condition added by the 5.6.40 merge
When Oracle fixed MDEV-13899 in their own way, they moved the
condition to the only caller of PageConverter::update_records().
Thus, the merge of 5.6.40 into MariaDB added a redundant condition.

PageConverter::update_records(): Move the page_is_leaf() condition
to the only caller, PageConverter::update_index_page().
2018-05-11 14:02:13 +03:00
Sergei Petrunia
c686483264 MDEV-12427: rocksdb.write_sync fails
Enable the test as the cause of the failure has been fixed
2018-05-11 13:56:42 +03:00