page_cur_dtuple_cmp(): Take DESC index fields into account also when
comparing a NULL value in a ROW_FORMAT≠REDUNDANT record.
Thanks to Elena Stepanova for finding this bug.
page_cur_search_with_match(): Remove rec_get_offsets(), and instead
determine the start and end of each field while comparing.
page_dir_slot_get_rec(), page_dir_slot_get_rec_validate():
Add a parameter to avoid invoking page_align().
page_cur_dtuple_cmp(): Replaces cmp_dtuple_rec_leaf() for both
leaf and non-leaf pages. In SPATIAL INDEX, non-leaf records are
special, because the child page number may be part of the comparison.
Reviewed by: Vladislav Lesin
For some reason, page_cur_search_with_match_bytes(), which can speed
up append operations (PAGE_CUR_LE used by INSERT), was only enabled
if innodb_adaptive_hash_index=ON even though it has nothing to do with
the adaptive hash index.
Furthermore, mysql/mysql-server@c9bbc83d11
a.k.a. commit c9bbc83d11 reduced a limit
from 3 to 2 but forgot to adjust the PAGE_N_DIRECTION limit accordingly.
We are adjusting that as well.
Reviewed by: Vladislav Lesin
During a workload, an adaptive hash index had been built on
UNIQUE INDEX(ID) on SYS_TABLES, and during a DROP TABLE
operation the adaptive hash index would be widened to cover
also the PRIMARY KEY(NAME) field that the index includes: (ID,NAME).
Such an adaptive hash index is unlikely to satisfy (m)any queries.
Let us limit the AHI prefix to the unique fields.
Reviewed by: Vladislav Lesin
btr_search_drop_page_hash_index(): Replace the Boolean parameter
with const dict_index_t *not_garbage. If buf_block_t::index points
to that, there is no need to acquire btr_sea::partition::latch.
The old parameter bool garbage_collect=false is equivalent to the
parameter not_garbage=nullptr. The parameter garbage_collect=true
will be replaced either with the actual index that is associated
with the buffer page, or with a bogus pointer not_garbage=-1 to
indicate that any lazily entries for a freed index need to be removed.
buf_page_get_low(), buf_page_get_gen(), mtr_t::page_lock(),
mtr_t::upgrade_buffer_fix(): Do not invoke
btr_search_drop_page_hash_index(). Our caller will have to do it
when appropriate.
buf_page_create_low(): Keep invoking btr_search_drop_page_hash_index().
This is the normal way of lazily dropping the adaptive hash index
after a DDL operation such as DROP INDEX operation.
btr_block_get(), btr_root_block_get(), btr_root_adjust_on_import(),
btr_read_autoinc_with_fallback(), btr_cur_instant_init_low(),
btr_cur_t::search_leaf(), btr_cur_t::pessimistic_search_leaf(),
btr_pcur_optimistic_latch_leaves(), dict_stats_analyze_index_below_cur():
Invoke btr_search_drop_page_hash_index(block, index) for pages that
may be leaf pages. No adaptive hash index may have been created on
anything else than a B-tree leaf page.
btr_cur_search_to_nth_level(): Do not invoke
btr_search_drop_page_hash_index(), because we are only accessing
non-leaf pages and the adaptive hash index may only have been created
on leaf pages.
btr_page_alloc_for_ibuf() and many other callers of buf_page_get_gen()
or similar functions do not invoke btr_search_drop_page_hash_index(),
because the adaptive hash index is never created on such pages.
If a page in the tablespace was freed as part of a DDL operation and
reused for something else, then buf_page_create_low() will take care
of dropping the adaptive hash index before the freed page will be
modified.
It is notable that while the flst_ functions may access pages that are
related to allocating B-tree index pages (the BTR_SEG_TOP and BTR_SEG_LEAF
linked from the index root page), those pages themselves can never be
stored in the adaptive hash index. Therefore, it is not necessary to
invoke btr_search_drop_page_hash_index() on them.
Reviewed by: Vladislav Lesin
btr_search_info_update_hash(): Do nothing if the record is positioned
on the page supremum or infimum pseudo-record. The adaptive hash index
can only include user records. This deficiency would cause the
adaptive hash index parameters to change between hashing a prefix of
1 field or a prefix of 1 byte.
Reviewed by: Vladislav Lesin
btr_search_guess_on_hash(): Only set BTR_CUR_HASH_FAIL on actual mismatch.
If the page latch cannot be acquired, the hash search might very well
have succeeded. Do not count that as a failure, that is, do not
unnecessarily invoke btr_search_update_hash_ref() after a normal search.
Set cursor->flag=BTR_CUR_HASH_ABORT if the current parameters of the
adaptive hash index are not suitable for the search and a call to
btr_cur_t::search_info_update() might help.
btr_cur_t::search_leaf(): Do not invoke search_info_update()
if btr_search_guess_on_hash() failed due to contention.
btr_cur_t::pessimistic_search_leaf(): Do not invoke search_info_update()
on the change buffer tree. Preivously, this condition was being checked
inside search_info_update().
btr_cur_t::search_leaf(): Do not attempt to use the adaptive
hash index for PAGE_CUR_G or PAGE_CUR_L, because those modes
expect an inequal result, and the adaptive hash index can only
deliver equal results.
btr_cur_t::check_mismatch(): Only handle PAGE_CUR_LE and PAGE_CUR_GE.
For PAGE_CUR_LE (bool ge=false), qualify a full match for the last
record of a page that is not at the end of the index. Previously,
an adaptive hash index lookup would fail when the record is at the end
of an index page but not at the end of the index. This would lead to
unnecessary rebuild of the adaptive hash index in read-only workloads.
Reviewed by: Vladislav Lesin
Now that ut_fold_ulint_pair() and ut_fold_binary() are no longer needed
for anything else than compatibility with old InnoDB data files that may
use innodb_checksum_algorithm=innodb, let us move the code to a single
compilation unit.
Reviewed by: Vladislav Lesin
For the adaptive hash index, dtuple_fold() and rec_fold() were employing
a slow rolling hash algorithm, computing hash values ("fold") for one
field and one byte at a time, while depending on calls to
rec_get_offsets().
We already have optimized implementations of CRC-32C and have been
successfully using that function in some other InnoDB tables, but not
yet in the adaptive hash index.
Any linear function such as any CRC will fail the avalanche test that
any cryptographically secure hash function is expected to pass:
any single-bit change in the input key should affect on average half
the bits in the output.
But we always were happy with less than cryptographically secure:
in fact, ut_fold_ulint_pair() or ut_fold_binary() are just about as
linear as any CRC, using a combination of multiplication and addition,
partly carry-less. It is worth noting that exclusive-or corresponds to
carry-less subtraction or addition in a binary Galois field, or GF(2).
We only need some way of reducing key prefixes into hash values.
The CRC-32C should be better than a Rabin–Karp rolling hash algorithm.
Compared to the old hash algorithm, it has the drawback that there will
be only 32 bits of entropy before we choose the hash table cell by a
modulus operation. The size of each adaptive hash index array is
(innodb_buffer_pool_size / 512) / innodb_adaptive_hash_index_parts.
With the maximum number of partitions (512), we would not exceed 1<<32
elements per array until the buffer pool size exceeds 1<<50 bytes (1 PiB).
We would hit other limits before that: the virtual address space on many
contemporary 64-bit processor implementations is only 48 bits (256 TiB).
So, we can simply go for the SIMD accelerated CRC-32C.
rec_fold(): Take a combined parameter n_bytes_fields. Determine the
length of each field on the fly, and compute CRC-32C over a single
contiguous range of bytes, from the start of the record payload area
to the end of the last full or partial field. For secondary index records
in ROW_FORMAT=REDUNDANT, also the data area that is reserved for NULL
values (to facilitate in-place updates between NULL and NOT NULL values)
will be included in the count. Luckily, InnoDB always zero-initialized
such unused area; refer to data_write_sql_null() in
rec_convert_dtuple_to_rec_old(). For other than ROW_FORMAT=REDUNDANT,
no space is allocated for NULL values, and therefore the CRC-32C will
only cover the actual payload of the key prefix.
dtuple_fold(): For ROW_FORMAT=REDUNDANT, include the dummy NULL values
in the CRC-32C, so that the values will be comparable with rec_fold().
innodb_ahi-t: A unit test for rec_fold() and dtuple_fold().
btr_search_build_page_hash_index(), btr_search_drop_page_hash_index():
Use a fixed-size stack buffer for computing the fold values, to avoid
dynamic memory allocation.
btr_search_drop_page_hash_index(): Do not release part.latch if we
need to invoke multiple batches of rec_fold().
dtuple_t: Allocate fewer bits for the fields. The maximum number of
data fields is about 1023, so uint16_t will be fine for them. The
info_bits is stored in less than 1 byte.
ut_pair_min(), ut_pair_cmp(): Remove. We can actually combine and compare
int(n_fields << 16 | n_bytes).
PAGE_CUR_LE_OR_EXTENDS, PAGE_CUR_DBG: Remove. These were never defined,
because they would only work with latin1_swedish_ci if at all.
btr_cur_t::check_mismatch(): Replaces !btr_search_check_guess().
cmp_dtuple_rec_bytes(): Replaces cmp_dtuple_rec_with_match_bytes().
Determine the offsets of fields on the fly.
page_cur_try_search_shortcut_bytes(): This caller of
cmp_dtuple_rec_bytes() will not be invoked on the change buffer tree.
cmp_dtuple_rec_leaf(): Replaces cmp_dtuple_rec_with_match()
for comparing leaf-page records.
buf_block_t::ahi_left_bytes_fields: Consolidated Atomic_relaxed<uint32_t>
of curr_left_side << 31 | curr_n_bytes << 16 | curr_n_fields.
The other set of parameters (n_fields, n_bytes, left_side) was removed
as redundant.
btr_search_update_hash_node_on_insert(): Merged to
btr_search_update_hash_on_insert().
btr_search_build_page_hash_index(): Take combined left_bytes_fields
instead of n_fields, n_bytes, left_side.
btr_search_update_block_hash_info(), btr_search_update_hash_ref():
Merged to btr_search_info_update_hash().
btr_cur_t::n_bytes_fields: Replaces n_bytes << 16 | n_fields.
We also remove many redundant checks of btr_search.enabled.
If we are holding any btr_sea::partition::latch, then a nonnull pointer
in buf_block_t::index must imply that the adaptive hash index is enabled.
Reviewed by: Vladislav Lesin
Let us use implement a simple fixed-size allocator for the adaptive hash
index, insted of complicating mem_heap_t or mem_block_info_t.
MEM_HEAP_BTR_SEARCH: Remove.
mem_block_info_t::free_block(), mem_heap_free_block_free(): Remove.
mem_heap_free_top(), mem_heap_get_top(): Remove.
btr_sea::partition::spare: Replaces mem_block_info_t::free_block.
This keeps one spare block per adaptive hash index partition, to
process an insert.
We must not wait for buf_pool.mutex while holding
any btr_sea::partition::latch. That is why we cache one block for
future allocations. This is protected by a new
btr_sea::partition::blocks_mutex in order to relieve pressure on
btr_sea::partition::latch.
btr_sea::partition::prepare_insert(): Replaces
btr_search_check_free_space_in_heap().
btr_sea::partition::erase(): Replaces ha_search_and_delete_if_found().
btr_sea::partition::cleanup_after_erase(): Replaces the most part of
ha_delete_hash_node(). Unlike the previous implementation, we will
retain a spare block for prepare_insert().
This should reduce some contention on buf_pool.mutex.
btr_search.n_parts: Replaces btr_ahi_parts.
btr_search.enabled: Replaces btr_search_enabled. This must hold
whenever buf_block_t::index is set while a thread is holding a
btr_sea::partition::latch.
dict_index_t::search_info: Remove pointer indirection, and use
Atomic_relaxed or Atomic_counter for most fields.
btr_search_guess_on_hash(): Let the caller ensure that latch_mode is
BTR_MODIFY_LEAF or BTR_SEARCH_LEAF. Release btr_sea::partition::latch
before buffer-fixing the block. The page latch that we already acquired
is preventing buffer pool eviction. We must validate both
block->index and block->page.state while holding part.latch
in order to avoid race conditions with buffer page relocation
or buf_pool_t::resize().
btr_search_check_guess(): Remove the constant parameter
can_only_compare_to_cursor_rec=false.
ahi_node: Replaces ha_node_t.
This has been tested by running the regression test suite
with the adaptive hash index enabled:
./mtr --mysqld=--loose-innodb-adaptive-hash-index=ON
Reviewed by: Vladislav Lesin
Add a simple test to verify the server behaves in a safe manner if configured
with ciphers that aren't compatible with the server certificate.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
Add a simple test to verify that the server will fail to start up when no valid
cipher suites are passed to `ssl-cipher`.
As different TLS libraries and versions have differing cipher suite support, it
would be a good idea to ensure the server behaves in a safe manner if it is
configured with invalid cipher suites.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
during FLUSH PRIVILEGES, allow_all_hosts temporarily goes out of sync
with acl_check_hosts and acl_wild_hosts.
As it's tested in acl_check_host() without a mutex, let's re-test it
under a mutex to make sure the value is correct.
Note that it's just an optimization and it's ok to see outdated
allow_all_hosts value here.
* replace the message away in the test result
* remove "feedback plugin:" prefix, it's a server message, not plugin's
* downgrade to the warning, because
1) it's not a failure, no operation was aborted, server still works
2) it's something actionable, so not a [Note] either
- InnoDB fails to recover the full crc32 encrypted page from
doublewrite buffer. The reason is that buf_dblwr_t::recover()
fails to identify the space id from the page because the page has
been encrypted from FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION bytes.
Fix:
===
buf_dblwr_t::recover(): preserve any pages whose space_id
does not match a known tablespace. These could be encrypted pages
of tablespaces that had been created with
innodb_checksum_algorithm=full_crc32.
buf_page_t::read_complete(): If the page looks corrupted and the
tablespace is encrypted and in full_crc32 format, try to
restore the page from doublewrite buffer.
recv_dblwr_t::recover_encrypted_page(): Find the page which
has the same page number and try to decrypt the page using
space->crypt_data. After decryption, compare the space id.
Write the recovered page back to the file.
Mariabackup (mariadb-backup) supports the --use-memory option that
sets the buffer pool size for innodb. However, current SST scripts
do not use this option. This commit adds support for this option,
the value for which can be specified via the "use_memory" parameter
in the configuration file in the [sst], [mariabackup] or [xtrabackup]
sections (supported only for compatibility with old configurations).
In addition, if the innodb_buffer_pool_size option is specified in
the user configuration (in the main server configuration sections)
or passed to the SST scripts or the server via arguments, its value
is also passed to mariadb-backup as the value for the --use-memory
option.
A new section name [mariabackup] has also been added, which can
be used instead of the deprecated [xtrabackup] (the section name
"mariabackup" was specified in the documentation, but was not
actually supported by SST scripts before this commit).
CURRENT_TEST: binlog_encryption.rpl_parallel_gco_wait_kill
mysqltest: In included file "./suite/rpl/t/rpl_parallel_gco_wait_kill.test":
included from /home/buildbot/amd64-ubuntu-2004-debug/build/mysql-test/suite/binlog_encryption/rpl_parallel_gco_wait_kill.test at line 2:
At line 334: Can't initialize replace from 'replace_result $thd_id THD_ID'
An sql thread can reach the "Slave has read all relay log" state
and then start reading relay log again. Let's use a more generic
pattern to retrieve the sql thread ID even if it's not
in the "read all relay log" state.
MDEV-29533 Crash when MariaDB is replica of MySQL 8.0
MySQL 8.0 has added the following new events in the MySQL binary log
PARTIAL_UPDATE_ROWS_EVENT
TRANSACTION_PAYLOAD_EVENT
HEARTBEAT_LOG_EVENT_V2
- PARTIAL_UPDATE_ROWS_EVENT is used by MySQL to generate update
statements using JSON_SET, JSON_REPLACE and JSON_REMOVE to make
update of JSON columns more efficient. These events can be
disabled by setting 'binlog-row-value-options=""'
- TRANSACTION_PAYLOAD_EVENT is used by MySQL to signal that a
row event is compressed. It an be disably by setting
'binlog_transaction_compression=0'.
- HEARTBEAT_LOG_EVENT_V2 is written to the binary log many times
per seconds. It can be ignored by the server.
What this patch does:
- If PARTIAL_UPDATE_ROWS_EVENT or TRANSACTION_PAYLOAD_EVENT is found,
the server will stop with an error message of how to disable the
MySQL server to generate such events.
- HEARTBEAT_LOG_EVENT_V2 events are ignored.
- mariadb-binlog will write the name of the new events.
- mariadb-binlog will stop if PARTIAL_UPDATE_ROWS_EVENT or
TRANSACTION_PAYLOAD_EVENT is found, unless --force is given.
- Fixes a crash in mariadb-binlog if a character set unknown to
MariaDB is found. (MDEV-29533)
From Kristian Nielsen:
- Add test case for MySQL 8.0 to MariaDB replication and fixed a
a small typo in post_header_len initialization.
Reviewer: knielsen@mariadb.org
This shows the maximum memory allocations used by the current connection.
The value for @@global.max_memory_used is 0 as we are not collecting this
value as it would cause a notable performance issue registering this for
all threads for every memory allocation
Reviewed-by: Sergei Golubchik <serg@mariadb.org>
In particular ""Read semi-sync reply magic number error" now prints out
what was wrong with the packet.
Reviewed-by: Sergei Golubchik <serg@mariadb.org>
The test with memory restrictions randomly works or fails in buildbot
depending on server configurations. On my machine the original test
worked.
As the test was there to just check if the server crashes when run with
small memory configurations, I disabled testing if the query would fail
or not. The test still has its original purpose.
Discussed with: Sergei Golubchik <serg@mariadb.org>
This commit updates default memory allocations size used with MEM_ROOT
objects to minimize the number of calls to malloc().
Changes:
- Updated MEM_ROOT block sizes in sql_const.h
- Updated MALLOC_OVERHEAD to also take into account the extra memory
allocated by my_malloc()
- Updated init_alloc_root() to only take MALLOC_OVERHEAD into account as
buffer size, not MALLOC_OVERHEAD + sizeof(USED_MEM).
- Reset mem_root->first_block_usage if and only if first block was used.
- Increase MEM_ROOT buffers sized used by my_load_defaults, plugin_init,
Create_tmp_table, allocate_table_share, TABLE and TABLE_SHARE.
This decreases number of malloc calls during queries.
- Use a small buffer for THD->main_mem_root in THD::THD. This avoids
multiple malloc() call for new connections.
I tried the above changes on a complex select query with 12 tables.
The following shows the number of extra allocations that where used
to increase the size of the MEM_ROOT buffers.
Original code:
- Connection to MariaDB: 9 allocations
- First query run: 146 allocations
- Second query run: 24 allocations
Max memory allocated for thd when using with heap table: 61,262,408
Max memory allocated for thd when using Aria tmp table: 419,464
After changes:
Connection to MariaDB: 0 allocations
- First run: 25 allocations
- Second run: 7 allocations
Max memory allocated for thd when using with heap table: 61,347,424
Max memory allocated for thd when using Aria table: 529,168
The new code uses slightly more memory, but avoids memory fragmentation
and is slightly faster thanks to much fewer calls to malloc().
Reviewed-by: Sergei Golubchik <serg@mariadb.org>
Heap tables are allocated blocks to store rows according to
my_default_record_cache (mapped to the server global variable
read_buffer_size).
This causes performance issues when the record length is big
(> 1000 bytes) and the my_default_record_cache is small.
Changed to instead split the default heap allocation to 1/16 of the
allowed space and not use my_default_record_cache anymore when creating
the heap. The allocation is also aligned to be just under a power of 2.
For some test that I have been running, which was using record length=633,
the speed of the query doubled thanks to this change.
Other things:
- Fixed calculation of max_records passed to hp_create() to take
into account padding between records.
- Updated calculation of memory needed by heap tables. Before we
did not take into account internal structures needed to access rows.
- Changed block sized for memory_table from 1 to 16384 to get less
fragmentation. This also avoids a problem where we need 1K
to manage index and row storage which was not counted for before.
- Moved heap memory usage to a separate test for 32 bit.
- Allocate all data blocks in heap in powers of 2. Change reported
memory usage for heap to reflect this.
Reviewed-by: Sergei Golubchik <serg@mariadb.org>
Note: Changes to the test innodb.stats_persistent
in commit e5c4c0842d (MDEV-35443)
are not merged, because the test scenario is impossible
due to commit e66928ab28 (MDEV-33462).
Problem:
=======
- insert..select statement on partition table fails to use
bulk insert for the transaction.
Solution:
========
- Enable the bulk insert operation for insert..select
statement for partition table.
