Commit graph

22789 commits

Author SHA1 Message Date
Marko Mäkelä
ca8c3be47d Merge 10.4 into 10.5 2020-01-03 16:15:40 +02:00
Oleksandr Byelkin
9d036f840a Merge branch '10.3' into 10.4 2020-01-03 15:05:50 +01:00
Oleksandr Byelkin
7753a29064 Merge branch '10.2' into 10.3 2020-01-03 13:44:16 +01:00
Marko Mäkelä
9949ab9393 MDEV-12353 preparation: Cleanup MLOG_FILE_NAME logging
mtr_t::commit_files(): Renamed from mtr_t::commit_checkpoint().
Remove the redundant bool parameter, and instead use checkpoint_lsn=0
to indicate that no checkpoint marker should be written.
2020-01-02 11:15:04 +02:00
Eugene Kosov
562c037b48 MDEV-18115 Remove dummy tablespace for the redo log
Redo log subsystem was decoupled from tablespace subsystem. It now manages file
descriptors for redo log files by itself.

FIL_TYPE_LOG: removed, code in various places was simplified

SRV_LOG_SPACE_FIRST_ID: renamed to SRV_SPACE_ID_UPPER_BOUND
  to better match its purpose. Code in various places was simplified

fil_n_log_flushes: replaced with log_sys::flushes
fil_n_pending_log_flushes: replaced with log_sys::pending_flushes

log_t::files::files: redo log file descriptors
log_t::files::file_names: redo log file names

log_t::files::set_file_names(): set file names without opening them
log_t::files::open_files(): opens redo log files
log_t::files::read(): treats several files as one big
log_t::files::write(): treats several files as one big
log_t::files::fsync(): flushes page cache to disk
log_t::files::close_files(): closes redo log files

fil_open_log_and_system_tablespace_files(): renamed to
  fil_open_system_tablespace_files()
  and obviously it now doesn't open redo log files

global files[1000]: removed. Why it was needed at all?
2020-01-01 22:09:51 +08:00
Marko Mäkelä
59d4f2a373 Cleanup after MDEV-12026
buf_flush_init_for_writing(): If innodb_checksum_algorithm=full_crc32,
return even earlier.

The logic to reset garbage FIL_PAGE_TYPE fields applies to
data pages in old data files that were last written before
commit 3926673ce7 was introduced.

When innodb_checksum_algorithm=full_crc32 is in effect, it only
applies to newly created data files. A file that was created in
full_crc32 format will keep that format. Other files will keep
some other format than full_crc32. Hence, the garbage-resetting
logic does not apply full_crc32 files.
2019-12-30 18:59:02 +02:00
Marko Mäkelä
4a012ce2f4 Post-fix for MDEV-12253: Remove redundant log writes
The 8 bytes at FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION
will be overwritten at page flush, in buf_page_encrypt_before_write(),
ever since commit 765a43605a
(MariaDB 10.1.23, 10.2.6, 10.3.1).

btr_store_big_rec_extern_fields(): Remove useless writes to the
page header (and to the redo log) for ROW_FORMAT=COMPRESSED BLOB pages.
2019-12-30 18:12:55 +02:00
Marko Mäkelä
3fa4a9e6be Merge 10.4 into 10.5 2019-12-30 10:29:43 +02:00
Marko Mäkelä
ffc0a08d05 Merge 10.3 into 10.4 2019-12-30 10:27:59 +02:00
Marko Mäkelä
02e3006957 MDEV-21405 Assertion failed on instant ADD COLUMN
btr_cur_pessimistic_insert(): Relax a too strict debug assertion that
would fail when the function is invoked by btr_cur_pessimistic_update()
during innobase_add_instant_try(), that is, when updating the hidden
metadata record during a subsequent ADD COLUMN operation involves
splitting the leftmost clustered index leaf page.

This is a partial backport of 301bd62b25
from 10.4.
2019-12-30 10:08:18 +02:00
Nikita Malyavin
4923604ee2 MDEV-18865 Assertion `t->first->versioned_by_id()' failed in innodb_prepare_commit_versioned
Cause:
* row_start != 0 treated as it exists. Probably, possible row permutations had not been taken in mind.

Solution:
* Checking both row_start and row_end is correct, so versioned() function is used
2019-12-29 12:16:04 +02:00
Marko Mäkelä
b36154a109 Cleanup log_rec_t 2019-12-27 21:20:03 +02:00
Marko Mäkelä
8cc15c036d Merge 10.4 into 10.5 2019-12-27 21:17:16 +02:00
Marko Mäkelä
b86e0f25f8 Cleanup: Use more page_id_t in crash recovery 2019-12-27 20:44:34 +02:00
Marko Mäkelä
4c25e75ce7 Merge 10.3 into 10.4 2019-12-27 18:20:28 +02:00
Marko Mäkelä
5ab70e7f68 Merge 10.2 into 10.3 2019-12-27 15:14:48 +02:00
Marko Mäkelä
16bce0f6fe Cleanup: Remove dict_delete_tablespace_and_datafiles()
The function was only called by innobase_drop_tablespace(),
which was removed in commit 494e4b99a4
and added in commit 2e814d4702.
2019-12-27 11:23:28 +02:00
Alexander Barkov
4c57ab34d4 Merge remote-tracking branch 'origin/10.3' into 10.4 2019-12-25 13:33:28 +04:00
Eugene Kosov
373443903b MDEV-21362 do something with -fno-builtin-memcmp for rem0cmp.cc
Such CMake check is not relevant as the oldest supported gcc is 4.8 and
compiler issue was solved in gcc-4.6 already.
https://godbolt.org/z/7G64qo

cmp_data(): simplify code
2019-12-24 17:06:04 +08:00
Eugene Kosov
cbc170adf2 MDEV-21362 do something with -fno-builtin-memcmp for rem0cmp.cc
innodb.cmake: restrict -fno-builtin-memcmp for GCC versions older that 4.6
where optimization issue was solved.
2019-12-24 17:03:27 +08:00
Thirunarayanan Balathandayuthapani
90ba87cb9e MDEV-19176 Reduce the memory usage during recovery
- post-push to fix the compilation issue
2019-12-23 15:56:57 +05:30
Thirunarayanan Balathandayuthapani
bba59abb03 MDEV-19176 Reduce the memory usage during recovery
- Moved the recv_sys->heap memory condition inside recv_parse_log_recs().
So that, InnoDB can mark the status as STORE_NO earlier.

- InnoDB uses one third of buffer pool chunk size for reading the redo
log records. In that case, we can avoid the scenario where buffer ran
out of memory issue during recovery.
2019-12-23 15:51:02 +05:30
Marko Mäkelä
a59a015a75 Plug memory leaks from numa_get_mems_allowed() 2019-12-23 07:47:16 +02:00
Marko Mäkelä
73985d8301 Merge 10.1 into 10.2 2019-12-23 07:14:51 +02:00
Eugene Kosov
496532b5c5 MDEV-20950: Fix 32-bit Windows build 2019-12-21 21:36:25 +02:00
Alexey Botchkov
9dadfdcde5 MDEV-14024 PCRE2.
Related changes in the server code.
2019-12-21 10:34:02 +01:00
Sergei Golubchik
aade6e53d3 fix a bad merge
in 10.1+ one should use

MY_CHECK_AND_SET_COMPILER_FLAG("-Wno-address-of-packed-member")

and it's already done in storage/tokudb/PerconaFT/CMakeLists.txt
2019-12-20 19:47:31 +01:00
Marko Mäkelä
8174e68895 MDEV-21371 Assertion failure in page_rec_get_next_low() during innodb_gis.rtree_compress
A debug assertion that was added in
commit ed0793e096
turns out to be too strict. In the test innodb_gis.rtree_compress,4k
the function is sometimes being invoked by purge for a
spatial index root page that is not a leaf page (PAGE_LEVEL is 1).
2019-12-20 19:21:37 +02:00
Marko Mäkelä
ce70573f62 MDEV-21353 Assertion failures due to btr_pcur_restore_pos() misbehaving
In commit befde6e97e
a bogus condition was added, aiming to avoid debug assertion failures
during ALTER TABLE...IMPORT TABLESPACE.

page_cur_delete_rec(): Remove the bogus condition, and replace the
use of buf_block_modify_clock_inc() with a lower-level operation
that skips the debug checks that would fail during IMPORT.
2019-12-20 17:57:32 +02:00
Eugene Kosov
f6b58d2916 MDEV-21239 ASAN use-after-poison in a server shutdown in innodb.innodb_buffer_pool_resize
If we manually poison memory region, ASAN requires that we manually unpoison
it too. We didn't do it and munmap() didn't do it too. That was the issue.

os_mem_free_large(): unpoison memory region. Just in case.

https://github.com/google/sanitizers/issues/182
86acaa9457/compiler-rt/include/sanitizer/asan_interface.h (L21)
2019-12-20 19:59:58 +08:00
Eugene Kosov
305081a735 check I/O buffer size and alignment in InnoDB 2019-12-19 23:46:49 +08:00
Eugene Kosov
a5a433e256 fix windows compilation 2019-12-18 23:35:33 +08:00
Marko Mäkelä
44be8652c5 Cleanup: Remove fil_space_get_flags()
Replace fil_space_get_flags(space) == ULINT_UNDEFINED
with the functionally equivalent fil_space_get_size(space) == 0.
2019-12-18 16:27:26 +02:00
Eugene Kosov
3d3d8f1012 MDEV-21337 fix aligned_malloc()
do not fallback to malloc(), always return properly aligned buffer
2019-12-18 20:09:52 +08:00
Sergei Golubchik
0d70f40bdb Disable -Werror for rocksdb submodule
We cannot have -Werror for third-party submodules because
whenever they break the build we cannot fix it as needed.
2019-12-17 21:22:57 +01:00
Marko Mäkelä
fb4a897fd9 MDEV-12353 preparation: Remove UNIV_LOG_LSN_DEBUG
The debug instrumentation with the MLOG_LSN pseudo-record has not been
used for debugging for years. Let us remove this code now.
It would have to be removed as part of MDEV-12353 or MDEV-14425 anyway,
when implementing a new redo log file format.
2019-12-17 15:39:21 +02:00
Marko Mäkelä
59a088744d Remove unused mlog_catenate_ulint_compressed()
The function was only used by trx_undo_page_init_log()
(writing the MLOG_UNDO_INIT record), which was removed in
commit ccb3550221.
2019-12-16 13:40:00 +02:00
Marko Mäkelä
c24253d0fa MDEV-21174: Fix the WITH_INNODB_EXTRA_DEBUG build
One reference to PageConverter::m_page_zip_ptr was not adjusted
commit 745fd4b39f when the
data member was removed.
2019-12-16 13:14:05 +02:00
Alexander Barkov
3d98892232 Merge remote-tracking branch 'origin/5.5' into 10.1 2019-12-16 13:08:17 +04:00
Marko Mäkelä
28c89b7151 Merge 10.4 into 10.5 2019-12-16 07:47:17 +02:00
Marko Mäkelä
745fd4b39f MDEV-21174: Remove some mlog_write_initial_log_record_fast()
Pass buf_block_t* to more functions that write redo log.

page_zip_write_node_ptr(), page_zip_write_blob_ptr(),
page_zip_compress_write_log_no_data():
Take buf_block_t* as parameter, and do not tolerate mtr=NULL.

page_zip_compress(): Do not tolerate mtr=NULL.

page_zip_dir_insert(): Take page_cur_t* as parameter.

mlog_write_initial_log_record(): Remove. This function was unused.

RecIterator::remove(): Remove the redundant page_zip parameter.

PageConverter::m_page_zip_ptr: Remove.
2019-12-13 18:15:51 +02:00
Marko Mäkelä
2b5a269cb4 MDEV-21174: Clean up record insertion
page_cur_insert_rec_low(): Take page_cur_t* as a parameter,
and do not tolerate mtr=NULL.

page_cur_insert_rec_zip(): Do not tolerate mtr=NULL.
2019-12-13 18:15:51 +02:00
Marko Mäkelä
befde6e97e MDEV-12353 preparation: Clean up page_cur_delete_rec()
page_cur_delete_rec(): Do not tolerate mtr=NULL.

page_delete_rec(): Merge with the only caller, RecIterator::remove().

RecIterator::m_mtr: New data member: a dummy mini-transaction.
2019-12-13 18:15:35 +02:00
Marko Mäkelä
8fa759a576 Merge 10.3 into 10.4
We disable the MDEV-21189 test galera.galera_partition
because it times out.
2019-12-13 17:30:37 +02:00
Eugene Kosov
a9ea0056c7 MDEV-21133: use aligned memcpy in redo log and buffer pool 2019-12-13 21:03:50 +07:00
Sergei Golubchik
794911a27a tokudb: disable check_huge_pages_in_practice()
crashes on Debian 10
2019-12-13 11:24:23 +01:00
Sergei Golubchik
91c3d99804 tokudb: fix to compile with gcc 9.2.0 2019-12-13 11:24:23 +01:00
Marko Mäkelä
3466b47b0d Merge 10.2 into 10.3 2019-12-13 10:08:57 +02:00
Eugene Kosov
f0aa073f2b MDEV-20950 Reduce size of record offsets
offset_t: this is a type which represents one record offset.
It's unsigned short int.

a lot of functions: replace ulint with offset_t

btr_pcur_restore_position_func(),
page_validate(),
row_ins_scan_sec_index_for_duplicate(),
row_upd_clust_rec_by_insert_inherit_func(),
row_vers_impl_x_locked_low(),
trx_undo_prev_version_build():
  allocate record offsets on the stack instead of waiting for rec_get_offsets()
  to allocate it from mem_heap_t. So, reducing  memory allocations.

RECORD_OFFSET, INDEX_OFFSET:
  now it's less convenient to store pointers in offset_t*
  array. One pointer occupies now several offset_t. And those constant are start
  indexes into array to places where to store pointer values

REC_OFFS_HEADER_SIZE: adjusted for the new reality

REC_OFFS_NORMAL_SIZE:
  increase size from 100 to 300 which means less heap allocations.
  And sizeof(offset_t[REC_OFFS_NORMAL_SIZE]) now is 600 bytes which
  is smaller than previous 800 bytes.

REC_OFFS_SEC_INDEX_SIZE: adjusted for the new reality

rem0rec.h, rem0rec.ic, rem0rec.cc:
  various arguments, return values and local variables types were changed to
  fix numerous integer conversions issues.

enum field_type_t:
  offset types concept was introduces which replaces old offset flags stuff.
  Like in earlier version, 2 upper bits are used to store offset type.
  And this enum represents those types.

REC_OFFS_SQL_NULL, REC_OFFS_MASK: removed

get_type(), set_type(), get_value(), combine():
  these are convenience functions to work with offsets and it's types

rec_offs_base()[0]:
  still uses an old scheme with flags REC_OFFS_COMPACT and REC_OFFS_EXTERNAL

rec_offs_base()[i]:
  these have type offset_t now. Two upper bits contains type.
2019-12-13 00:26:50 +07:00
Eugene Kosov
014e125830 optimize crash recovery
recv_dblwr_t::list is used for appending to the beginning and iterating
through its elements. std::deque fits better for that purpose because
it does less allocations than std::forward_list and provides better memory
locality.
2019-12-12 22:19:41 +07:00