Commit graph

21240 commits

Author SHA1 Message Date
Marko Mäkelä
0460d42b94 Merge mariadb-10.2.33 2020-08-10 17:31:49 +03:00
Sergei Golubchik
fbcae42c2a Merge branch '10.1' into 10.2 2020-08-06 16:47:39 +02:00
Sergei Golubchik
a09a06d597 Merge branch 'merge/merge-tokudb-5.6' into 10.1 2020-08-05 01:46:02 +02:00
Sergei Golubchik
2adaaeba83 5.6.49-89.0 2020-08-04 12:44:43 +02:00
Marko Mäkelä
91caf130b7 MDEV-23101 fixup: Remove redundant code
lock_rec_has_to_wait_in_queue(): Remove an obviously redundant assertion
that was added in commit a8ec45863b
and also enclose a Galera-specific condition in #ifdef WITH_WSREP.
2020-08-04 09:56:09 +03:00
Jan Lindström
a8ec45863b MDEV-23101: SIGSEGV in lock_rec_unlock() when Galera is enabled
lock_rec_has_to_wait
wsrep_kill_victim
lock_rec_create_low
lock_rec_add_to_queue
DeadlockChecker::select_victim()

	THD can't change from normal transaction to BF (brute force) transaction
	here, thus there is no need to syncronize access in wsrep_thd_is_BF
	function.

lock_rec_has_to_wait_in_queue

	Add condition that lock is not NULL and add assertions if we are in
	strong state.
2020-08-03 15:15:40 +03:00
Sergei Golubchik
acfc500d03 compilation error on bintar-centos6-amd64-debug
/home/buildbot/buildbot/build/storage/xtradb/mtr/mtr0mtr.cc:97:37: error: invalid access to non-static data member ‘fil_space_t::latch’  of NULL object [-Werror=invalid-offsetof]
2020-08-03 09:48:52 +02:00
Oleksandr Byelkin
6d8af36bc7 Merge remote-tracking branch 'connect/10.2' into 10.2 2020-08-02 11:14:56 +02:00
Oleksandr Byelkin
ef7cb0a0b5 Merge branch '10.1' into 10.2 2020-08-02 11:05:29 +02:00
Thirunarayanan Balathandayuthapani
5ec40fbb27 MDEV-14711 Fix-up 2020-07-31 16:45:35 +05:30
Marko Mäkelä
879ba1979b MDEV-11799 Doublewrite recovery can corrupt data pages
The purpose of the InnoDB doublewrite buffer is to make InnoDB
tolerant against cases where the server was killed in the middle
of a page write. (In Linux, killing a process may interrupt a
write system call, typically on a 4096-byte boundary.)

There may exist multiple copies of a page number in the doublewrite
buffer. Recovery should choose the latest valid copy of the page.
By design, the FIL_PAGE_LSN must not precede the latest checkpoint LSN
nor be later than the end of the recovered log.

For page_compressed and encrypted pages, we were missing proper
consistency checks. In the 10.4 data set generated for in MDEV-23231,
the data file contained a valid page_compressed page, and an
identical copy of that page was also present in the doublewrite
buffer. But, recovery would incorrectly consider the page invalid
and restore an uncompressed copy of the same page that had been
written before the log checkpoint. (In fact, no redo log was to
be applied to that page.)

buf_dblwr_process(): Validate the FIL_PAGE_LSN in the doublewrite
buffer pages, and always skip page 0, because those pages should
have been recovered by Datafile::restore_from_doublewrite() if
necessary.

Datafile::restore_from_doublewrite(): Choose the latest applicable
page from the doublewrite buffer.

recv_dblwr_t::find_page(): Also validate encrypted or
page_compressed pages.

recv_dblwr_t::validate_page(): New function to validate a page,
either a copy in a data file or in the doublewrite buffer.
Also validate encrypted or page_compressed pages.

This is joint work with Thirunarayanan Balathandayuthapani.
2020-07-31 11:54:35 +03:00
Marko Mäkelä
f35d172103 MDEV-23198 Crash in REPLACE
row_vers_impl_x_locked_low(): clust_offsets may point to memory
that is allocated by mem_heap_alloc() and may have been freed.
For initializing clust_offsets, try to use the stack-allocated
buffer instead of a pointer that may point to freed memory.

This fixes a regression that was introduced in
commit f0aa073f2b (MDEV-20950).
2020-07-31 11:54:35 +03:00
Thirunarayanan Balathandayuthapani
8a612314d0 MDEV-23332 Index online status assert failure in btr_search_drop_page_hash_index
Problem:
========
In row_merge_drop_indexes(), InnoDB drops only the index from
dictionary and frees the index pages but it maintains the index
object if the table is being used by other DML threads. It sets
the online status of the index to ONLINE_INDEX_ABORTED_DROPPED.
Removing the index from dictionary doesn't remove the
corressponding ahi entries of the index. When block is being
reused, InnoDB tries to remove ahi entries for the block and
it fails if index online status is ONLINE_INDEX_ABORTED_DROPPED.

Fix:
====
MDEV-22456 allows the index ahi entries to be dropped lazily.
so checking online status in btr_search_drop_page_hash_index()
is meaningless and should be removed.
2020-07-30 13:59:03 +05:30
Marko Mäkelä
4860fe244b XtraDB 5.6.49-89.0
The only change between Percona XtraDB Server 5.6.48-88.0
and 5.6.49-89.0 (apart from the version number change) was
percona/percona-server@25ec240920
which we had already addressed in
commit 7c03edf2fe and
commit c0fca2863b.
2020-07-30 11:06:46 +03:00
Marko Mäkelä
c5d4dd2533 MDEV-23339 innodb_force_recovery=2 may still abort the rollback of recovered transactions
trx_rollback_active(), trx_rollback_resurrected(): Replace
an incorrect condition that we failed to replace in
commit b68f1d847f (MDEV-21217).
2020-07-30 09:24:36 +03:00
Sergei Golubchik
77b7f793f9 MDEV-18496 Crash when Aria encryption is enabled but plugin not available
fix uninitialized struct member
2020-07-29 14:56:24 +02:00
Sergei Golubchik
88cbe2f081 MDEV-18496 Crash when Aria encryption is enabled but plugin not available
wait_while_table_is_used() should return an error if handler::extra() fails
2020-07-29 14:56:24 +02:00
Marko Mäkelä
3c3f172f17 MDEV-23308 CHECK TABLE attempts to access parent_right_page_no=FIL_NULL
mysql/mysql-server@e00ad49edc
which introduced WL#6326 to MySQL 5.7.2 added a buffer page
acquisition to CHECK TABLE code (solely for the purpose of
obeying the changed latching order), but failed to check that
a parent page actually exists. It would not necessarily exist in a
corrupted index where a parent page is missing pointer records
to child pages.
2020-07-28 13:00:59 +03:00
Daniel Black
940668f5cb MDEV-23137: aarch64, postfix - cmake include 2020-07-28 19:53:19 +10:00
Daniel Black
cae4b3f811 rocksdb: FreeBSD disable jemalloc search
FreeBSD's inbuilt default jemalloc means its pointless
to do a package search on it. The paths are already set
by the system defaults.
2020-07-28 10:49:47 +10:00
Daniel Black
715beee46a MDEV-23051: riscv64 fails build (atomics)
riscv64 fails to build because the use
of #include <atomic> needs to link with -latomic.

per https://github.com/riscv/riscv-gnu-toolchain/issues/183#issuecomment-253721765
2020-07-28 10:13:45 +10:00
Krunal Bauskar
d88ea26088 MDEV-23137: RocksDB: undefined reference to crc32c_arm64
RocksDB fails to build on arm64: undefined reference to
            `crc32c_arm64(unsigned int, unsigned char const*, unsigned int)'

MariaDB uses storage/rocksdb/build_rocksdb.cmake to compile RocksDB.
Said cmake missed adding crc32c_arm64 compilation target so if
machine native architecture supported crc32 then complier would enable
usage of function defined in crc32c_arm64 causing the listed error.

Added crc32c_arm64 complition target.

closes 
2020-07-28 10:06:29 +10:00
Daniel Black
186d9d0d72 MDEV-12474: rocksdb: mtr - rocksdb.concurrent_alter use sh
FreeBSD doesn't have bash installed by default and sh
has sufficient job control for this test.

$  mysql-test/mtr --mem --max-test-fail=30 --force --parallel=1 rocksdb.concurrent_alter
Logging: /home/dan/mariadb-server-10.5/mysql-test/mysql-test-run.pl  --mem --max-test-fail=30 --force --parallel=1 rocksdb.concurrent_alter
vardir: /usr/home/dan/build-mariadb-server-10.5/mysql-test/var
Checking leftover processes...
Removing old var directory...
Creating var directory '/usr/home/dan/build-mariadb-server-10.5/mysql-test/var'...
 - symlinking 'var' to '/tmp/var_auto_P81m'
Checking supported features...
MariaDB Version 10.5.4-MariaDB
 - SSL connections supported

 - binaries built with wsrep patch
Collecting tests...
Installing system database...

==============================================================================

TEST                                      RESULT   TIME (ms) or COMMENT
--------------------------------------------------------------------------

worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
rocksdb.concurrent_alter 'write_committed' [ pass ]  16348
rocksdb.concurrent_alter 'write_prepared' [ pass ]  16771
--------------------------------------------------------------------------
The servers were restarted 1 times
Spent 33.119 of 41 seconds executing testcases

Completed: All 2 tests were successful.

$ uname -a
FreeBSD freebsd 12.1-RELEASE-p6 FreeBSD 12.1-RELEASE-p6 GENERIC  amd64
2020-07-28 09:02:29 +10:00
Thirunarayanan Balathandayuthapani
a1f899a8ab MDEV-23233: Race condition for btr_search_drop_page_hash_index() in buf_page_create()
commit ad6171b91c (MDEV-22456)
introduced code to buf_page_create() that would lazily drop
adaptive hash index entries for an index that has been
evicted from the data dictionary cache.

Unfortunately, that call was missing adequate protection.
While the btr_search_drop_page_hash_index(block) was executing,
the block could be reused for something else.

buf_page_create(): If btr_search_drop_page_hash_index() must be
invoked, pin the block before releasing the buf_pool->page_hash lock,
so that the block cannot be grabbed by other threads.
2020-07-27 22:17:51 +05:30
Thirunarayanan Balathandayuthapani
5f1ec5cbb7 MDEV-14711 Assertion `mode == 16 || mode == 12 || !fix_block->page.file_page_was_freed' failed in buf_page_get_gen (rollback requesting a freed undo page)
Problem:
=======
In buf_cur_optimistic_latch_leaves(), requesting a left block with BTR_GET
after releasing current block. But there is no guarantee that left block
could be still available.

Fix:
====

(1) In btr_cur_optimistic_latch_leaves(), replace the BUF_GET with
BUF_GET_POSSIBLY_FREED for fetching left block.
(2) Once InnoDB acquires left block, it should check FIL_PAGE_NEXT with
current block page number. If not, release cursor->left_block and return
false.
2020-07-24 20:32:27 +05:30
Teemu Ollakka
8ef41c6084 MDEV-23272 Galera stack-use-after-scope error with ASAN build
THD proc info was assigned from stack allocated temporary buffer
which went out of scope immediately after assignment.

Fixed by removing the use of temp buffer and assign proc info
from string literal.
2020-07-24 08:50:15 +03:00
Thirunarayanan Balathandayuthapani
b3b1c51e4d MDEV-23134 SEGV in dict_load_table_one during restart after server crash
Problem:
========
dict_load_table_one() doesn't handle the scenario where clustered
index page is FIL_NULL when DICT_ERR_IGNORE_INDEX_ROOT mode
is set.

Fix:
====
InnoDB should set the file_unreadable when it can't find the
clustered index root page.
2020-07-23 20:43:28 +05:30
Thirunarayanan Balathandayuthapani
adeb736f9a MDEV-22903 heap-use-after-free while accessing fts cache deleted doc ids
Problem:
=======
  fts_cache_append_deleted_doc_ids() holds the deleted_lock and tries to
access size of deleted_doc_ids. In the meantime, fts_cache_clear()
clears the sync_heap before clearing deleted_doc_ids. It leads to
invalid access of deleted_doc_ids.

Fix:
===
fts_cache_clear() should free the sync_heap after clearing
deleted_doc_ids.
2020-07-23 16:34:38 +05:30
Thirunarayanan Balathandayuthapani
fe39d02f51 MDEV-20638 Remove the deadcode from srv_master_thread() and srv_active_wake_master_thread_low()
- Due to commit fe95cb2e40 (MDEV-16125),
InnoDB master thread does not need to call srv_resume_thread()
and therefore there is no need to wake up the thread.
Due to the above patch, InnoDB should remove the following dead code.

srv_check_activity(): Makes the parameter as in,out and returns the
recent activity value

innobase_active_small(): Removed

srv_active_wake_master_thread(): Removed

srv_wake_master_thread(): Removed

srv_active_wake_master_thread_low(): Removed

Simplify srv_master_thread() and remove switch cases, added the assert.

Replace srv_wake_master_thread() with srv_inc_activity_count()

INNOBASE_WAKE_INTERVAL: Removed
2020-07-23 16:23:20 +05:30
Marko Mäkelä
52ccedd6dd MDEV-23268 SIGSEGV on srv_monitor_event if InnoDB is read-only
The srv_monitor_event and the srv_monitor_thread would not be
created when InnoDB is in read-only mode. Yet, some code would
unconditionally invoke os_event_set(srv_monitor_event).
2020-07-23 09:59:16 +03:00
Thirunarayanan Balathandayuthapani
3a8943ae73 MDEV-17481 mariadb service won't shutdown when it's running and the OS datetime updated backwards
__pthread_cond_timedwait() in page cleaner hangs if os time moved
backwards.Workaround could be waking up the page cleaner thread in
logs_empty_and_mark_files_at_shutdown(). But there is possibility that
server could hang when server is running. So InnoDB should wake up page
cleaner thread periodically in srv_master_do_idle_tasks().
2020-07-22 18:02:52 +05:30
Thirunarayanan Balathandayuthapani
2a3bc0b9cd MDEV-13830 Assertion failed: recv_sys->mlog_checkpoint_lsn <= recv_sys->recovered_lsn
There can be multiple MLOG_CHECKPOINT record for the same checkpoint.
During recovery, InnoDB could encounter the previous MLOG_CHECKPOINT
for the checkpoint lsn. So the assertion
mlog_checkpoint_lsn <= recovered_lsn is wrong.
2020-07-22 18:00:03 +05:30
Varun Gupta
62d73df6b2 MDEV-19232: Floating point precision / value comparison problem
The issue occurs when the subquery_cache is enabled.
When there is a cache miss the division was leading to a value with scale 9.
In the case of cache hit the value returned was of scale 9 and due to the different
values for the scales the where condition evaluated to FALSE, hence the output
was incomplete.

To fix this problem we need to round up the decimal to the limit mentioned in
Item::decimals. This would make sure the values are compared with the same
scale.
2020-07-22 14:44:25 +05:30
Marko Mäkelä
ca9276e37e Merge 10.1 into 10.2 2020-07-20 14:53:24 +03:00
Marko Mäkelä
57ec42bc32 MDEV-23190 InnoDB data file extension is not crash-safe
When InnoDB is extending a data file, it is updating the FSP_SIZE
field in the first page of the data file.

In commit 8451e09073 (MDEV-11556)
we removed a work-around for this bug and made recovery stricter,
by making it track changes to FSP_SIZE via redo log records, and
extend the data files before any changes are being applied to them.

It turns out that the function fsp_fill_free_list() is not crash-safe
with respect to this when it is initializing the change buffer bitmap
page (page 1, or generally, N*innodb_page_size+1). It uses a separate
mini-transaction that is committed (and will be written to the redo
log file) before the mini-transaction that actually extended the data
file. Hence, recovery can observe a reference to a page that is
beyond the current end of the data file.

fsp_fill_free_list(): Initialize the change buffer bitmap page in
the same mini-transaction.

The rest of the changes are fixing a bug that the use of the separate
mini-transaction was attempting to work around. Namely, we must ensure
that no other thread will access the change buffer bitmap page before
our mini-transaction has been committed and all page latches have been
released.

That is, for read-ahead as well as neighbour flushing, we must avoid
accessing pages that might not yet be durably part of the tablespace.

fil_space_t::committed_size: The size of the tablespace
as persisted by mtr_commit().

fil_space_t::max_page_number_for_io(): Limit the highest page
number for I/O batches to committed_size.

MTR_MEMO_SPACE_X_LOCK: Replaces MTR_MEMO_X_LOCK for fil_space_t::latch.

mtr_x_space_lock(): Replaces mtr_x_lock() for fil_space_t::latch.

mtr_memo_slot_release_func(): When releasing MTR_MEMO_SPACE_X_LOCK,
copy space->size to space->committed_size. In this way, read-ahead
or flushing will never be invoked on pages that do not yet exist
according to FSP_SIZE.
2020-07-20 14:48:56 +03:00
Marko Mäkelä
98e2c17e9e Cleanup: Remove fil_check_adress_in_tablespace() 2020-07-20 14:48:56 +03:00
Marko Mäkelä
14543afd59 Cleanup: Remove unused AbstractCallback::m_free_limit 2020-07-20 14:48:56 +03:00
Marko Mäkelä
147d4b1ec0 MDEV-21347 innodb_log_optimize_ddl=OFF is not crash safe
In commit 0f90728bc0 (MDEV-16809)
we introduced the configuration option innodb_log_optimize_ddl
for controlling whether native index creation or table-rebuild
in InnoDB should avoid writing full redo log.

Fungo Wang reported that this option is causing occasional failures.
The reason is that pages may be written to data files in an
inconsistent state. Applying log records to such inconsistent pages
may fail.

The solution is to always invoke PageBulk::finish() before page latches
may be released, to ensure that the page contents is in a consistent
state.

Something similar was implemented in MySQL 8.0.13:
mysql/mysql-server@d1254b9473

buf_block_t::skip_flush_check: Remove. Suppressing consistency checks
is a bad idea.

PageBulk::needs_finish(): New predicate: Determine whether
PageBulk::finish() must fix up the page.

PageBulk::init(): Clear PAGE_DIRECTION to ensure that needs_finish()
will hold. We change the field from PAGE_NO_DIRECTION to 0
and back without writing redo log. This trick avoids the need
to introduce any new data member to PageBulk.

PageBulk::insert(): Replace some high-level accessors to bypass
debug assertions related to PAGE_HEAP_TOP that we will be violating
until finish() has been executed.

PageBulk::finish(): Tolerate m_rec_no==0. We must invoke this also
on an empty page, to ensure that PAGE_HEAP_TOP is initialized.

PageBulk::commit(): Always invoke finish().

PageBulk::release(), BtrBulk::pageSplit(), BtrBulk::storeExt(),
BtrBulk::finish(): Invoke PageBulk::finish().
2020-07-16 06:35:15 +03:00
Marko Mäkelä
fee11c7727 Make page validation stricter
page_simple_validate_old(), page_simple_validate_new():
Require PAGE_N_DIR_SLOTS to be at least 2.
2020-07-15 19:41:01 +03:00
Marko Mäkelä
38b4c07833 MDEV-23183 Infinite loop on page_validate() on corrupted page
MDEV-22721 (commit eba2d10ac5)
inadvertently introduced an infinite loop.

page_validate(): Remove the infinite loop.
2020-07-15 19:41:01 +03:00
Marko Mäkelä
07e89bf7d1 MDEV-23163 Merge new release of InnoDB 5.7.31 to 10.2
The only InnoDB change between MySQL 5.7.30 and MySQL 5.7.31
that is applicable to MariaDB Server was applied in
commit 8d061996e6 (MDEV-23161).
2020-07-14 15:13:40 +03:00
Marko Mäkelä
646a6005e7 Merge 10.1 into 10.2 2020-07-14 15:10:59 +03:00
Marko Mäkelä
67a03b7c94 XtraDB 5.6.48-88.0
The only InnoDB changes between Percona XtraDB Server 5.6.47-87.0
and 5.6.48-88.0 are related to InnoDB changes between MySQL 5.6.47
and MySQL 5.6.48, which we had already applied.
2020-07-14 13:32:32 +03:00
Marko Mäkelä
142f85142a Update the InnoDB version number to 5.6.49
There were no InnoDB changes between MySQL 5.6.48 and MySQL 5.6.49.
2020-07-14 13:25:18 +03:00
Marko Mäkelä
8d061996e6 MDEV-23161 avg_count_reset may wrongly be NULL in I_S.INNODB_METRICS
This issue was originally reported by Fungo Wang, along with a fix, as
MySQL Bug .

His suggested fix was applied as part of
mysql/mysql-server@a003fc373d
and released in MySQL 5.7.31.

i_s_metrics_fill(): Add the missing call to Field::set_notnull(),
and simplify some code.
2020-07-14 13:21:01 +03:00
Thirunarayanan Balathandayuthapani
dc58987eb7 MDEV-22765 i_s_fts_index_cache_fill_one_index() is not protect by the lock
- i_s_fts_index_cache_fill() should take shared lock of fts cache
before accessing index cache to avoid reading stale data.
2020-07-14 14:26:49 +05:30
Thirunarayanan Balathandayuthapani
194a720e28 MDEV-22890 DEADLOCK of threads detected: row0sel.cc S-LOCK / btr0cur.cc S-LOCK / row0quiesce.cc X-LOCK
Problem:
=======
- Read operations are always allowed to hold a secondary index leaf
latch and then look up the corresponding clustered index record.
Flush table operation acquires secondary index latch while holding
a clustered index latch. It leads to deadlock violation.

Fix:
====
- Flush table operation should acquire secondary index before taking
clustered index to avoid deadlock violation with select operation.
2020-07-14 12:47:32 +05:30
Olivier Bertrand
ad0d2424dd Fix compile error for KVM. Modified filamdbf.cpp 2020-07-13 20:02:51 +02:00
Olivier Bertrand
1e07df99f3 - Fix MDEV-22561 Unable to access DBF inside a ZIP archive
modified:   storage/connect/filamdbf.cpp
  modified:   storage/connect/filamdbf.h
  modified:   storage/connect/filamzip.cpp
  modified:   storage/connect/filamzip.h
  modified:   storage/connect/ha_connect.cc
  modified:   storage/connect/plgxml.cpp
  modified:   storage/connect/tabdos.cpp
  modified:   storage/connect/tabdos.h
  modified:   storage/connect/tabfix.h

- Add/Init Level class member
  modified:   storage/connect/mongo.cpp
  modified:   storage/connect/mongo.h
  modified:   storage/connect/tabjson.cpp
  modified:   storage/connect/tabjson.h

- Typo
  modified:   storage/connect/connect.cc
2020-07-13 16:30:57 +02:00
Marko Mäkelä
838a1046b2 MDEV-20377: Fix cmake -DPLUGIN_PERFSCHEMA=NO 2020-07-02 06:03:59 +03:00