Commit graph

194174 commits

Author SHA1 Message Date
Marko Mäkelä
e0f7c89c18 MDEV-26747 after-merge fix for MDEV-12026, MDEV-25105 2021-11-19 09:10:40 +02:00
Marko Mäkelä
aaef2e1d8c MDEV-27058: Reduce the size of buf_block_t and buf_page_t
buf_page_t::frame: Moved from buf_block_t::frame.
All 'thin' buf_page_t describing compressed-only ROW_FORMAT=COMPRESSED
pages will have frame=nullptr, while all 'fat' buf_block_t
will have a non-null frame pointing to aligned innodb_page_size bytes.
This eliminates the need for separate states for
BUF_BLOCK_FILE_PAGE and BUF_BLOCK_ZIP_PAGE.

buf_page_t:🔒 Moved from buf_block_t::lock. That is, all block
descriptors will have a page latch. The IO_PIN state that was used
for discarding or creating the uncompressed page frame of a
ROW_FORMAT=COMPRESSED block is replaced by a combination of read-fix
and page X-latch.

page_zip_des_t::fix: Replaces state_, buf_fix_count_, io_fix_, status
of buf_page_t with a single std::atomic<uint32_t>. All modifications
will use store(), fetch_add(), fetch_sub(). This space was previously
wasted to alignment on 64-bit systems. We will use the following encoding
that combines a state (partly read-fix or write-fix) and a buffer-fix
count:

buf_page_t::NOT_USED=0 (previously BUF_BLOCK_NOT_USED)
buf_page_t::MEMORY=1 (previously BUF_BLOCK_MEMORY)
buf_page_t::REMOVE_HASH=2 (previously BUF_BLOCK_REMOVE_HASH)
buf_page_t::FREED=3 + fix: pages marked as freed in the file
buf_page_t::UNFIXED=1U<<29 + fix: normal pages
buf_page_t::IBUF_EXIST=2U<<29 + fix: normal pages; may need ibuf merge
buf_page_t::REINIT=3U<<29 + fix: reinitialized pages (skip doublewrite)
buf_page_t::READ_FIX=4U<<29 + fix: read-fixed pages (also X-latched)
buf_page_t::WRITE_FIX=5U<<29 + fix: write-fixed pages (also U-latched)
buf_page_t::WRITE_FIX_IBUF=6U<<29 + fix: write-fixed; may have ibuf
buf_page_t::WRITE_FIX_REINIT=7U<<29 + fix: write-fixed (no doublewrite)

buf_page_t::write_complete(): Change WRITE_FIX or WRITE_FIX_REINIT to
UNFIXED, and WRITE_FIX_IBUF to IBUF_EXIST, before releasing the U-latch.

buf_page_t::read_complete(): Renamed from buf_page_read_complete().
Change READ_FIX to UNFIXED or IBUF_EXIST, before releasing the X-latch.

buf_page_t::can_relocate(): If the page latch is being held or waited for,
or the block is buffer-fixed or io-fixed, return false. (The condition
on the page latch is new.)

Outside buf_page_get_gen(), buf_page_get_low() and buf_page_free(), we
will acquire the page latch before fix(), and unfix() before unlocking.

buf_page_t::flush(): Replaces buf_flush_page(). Optimize the
handling of FREED pages.

buf_pool_t::release_freed_page(): Assume that buf_pool.mutex is held
by the caller.

buf_page_t::is_read_fixed(), buf_page_t::is_write_fixed(): New predicates.

buf_page_get_low(): Ignore guesses that are read-fixed because they
may not yet be registered in buf_pool.page_hash and buf_pool.LRU.

buf_page_optimistic_get(): Acquire latch before buffer-fixing.

buf_page_make_young(): Leave read-fixed blocks alone, because they
might not be registered in buf_pool.LRU yet.

recv_sys_t::recover_deferred(), recv_sys_t::recover_low():
Possibly fix MDEV-26326, by holding a page X-latch instead of
only buffer-fixing the page.
2021-11-18 17:47:19 +02:00
Marko Mäkelä
db915f7387 MDEV-27058: Move buf_page_t::slot to IORequest::slot
MDEV-23855 and MDEV-23399 already moved some transient data fields
from buffer pool page descriptors to IORequest, but the write buffer
of PAGE_COMPRESSED or ENCRYPTED tables was missed. Since is only needed
during asynchronous page write requests, it belongs to IORequest.
2021-11-18 17:44:33 +02:00
Marko Mäkelä
02e72f7b44 MDEV-26769 follow-up: Remove unnecessary page locking
btr_cur_optimistic_latch_leaves(): Use transactional_shared_lock_guard.

btr_cur_latch_leaves(): Avoid acquiring some page latches, because
the changes are already blocked by index->lock.

btr_cur_search_to_nth_level_func(): Remove a redundant variable
retrying_for_search_prev=!!prev_tree_blocks, and avoid acquiring
some page latches.
2021-11-18 17:44:33 +02:00
Marko Mäkelä
14c5178f25 MDEV-27069: heap-use-after-free in dict_stats_recalc_pool_del()
dict_stats_recalc_pool_del(): Always reposition the iterators after
releasing and reacquiring the mutex. Another thread could have modified
recalc_pool, causing reallocation of the underlying memory while
we were waiting.

This fixes a regression that was caused by
commit 45a05fda27 (MDEV-25919).
2021-11-18 17:39:39 +02:00
Marko Mäkelä
862eccd524 MDEV-26769 fixup: Fix the SUX_LOCK_GENERIC build 2021-11-18 17:34:19 +02:00
Marko Mäkelä
0a168398a0 Merge 10.5 into 10.6 2021-11-17 15:03:47 +02:00
Marko Mäkelä
5489ce0ae1 Merge 10.4 into 10.5 2021-11-17 14:49:12 +02:00
Marko Mäkelä
70e788b1e5 Merge 10.3 into 10.4 2021-11-17 13:59:42 +02:00
Marko Mäkelä
9962cda527 Merge 10.2 into 10.3 2021-11-17 13:55:54 +02:00
Marko Mäkelä
878f7e38c7 MDEV-23805 fixup: Adjsut the MDEV-16131 and MDEV-24730 tests
MDEV-23805 simplified the treatment of empty tables during ALTER TABLE,
which could prevent the scenarios that were previously reported and
fixed as MDEV-16131 and MDEV-24730.

With the MDEV-23805 fix, the statement
SET DEBUG_SYNC = 'now WAIT_FOR copied';
could occasionally time out, depending on timing.
Apparently, there was a race condition where purge could resume
(and empty the table) before ALTER TABLE got the chance to execute.
We must prevent the purge of history from running before
ALTER TABLE has started executing.
2021-11-17 13:40:21 +02:00
Eugene Kosov
ed0a224b3d MDEV-26747 improve corruption check for encrypted tables on ALTER IMPORT
fil_space_decrypt(): change signature to return status via dberr_t only.
Also replace impossible condition with an assertion and prove it via
test cases.
2021-11-17 15:49:22 +06:00
Igor Babaev
8f24f5fee2 MDEV-26825 Bogus error for query with two usage of CTE referring another CTE
This bug affected queries with two or more references to a CTE referring
another CTE if the definition of the latter contained an invocation of
a stored function that used a base table. The bug could lead to a bogus
error message or to an assertion failure.
  For any non-first reference to CTE cte1 With_element::clone_parsed_spec()
is called that parses the specification of cte1 to construct the unit
structure for this usage of cte1. If cte1 refers to another CTE cte2
outside of the specification of cte1 then With_element::clone_parsed_spec()
has to be called for cte2 as well. This call is made by the function
LEX::resolve_references_to_cte() within the invocation of the function
With_element::clone_parsed_spec() for cte1.
  When the specification of a CTE is parsed all table references encountered
in it must be added to the global list of table references for the query.
As the specification for the non-first usage of a CTE is parsed at a
recursive call of the parser the function With_element::clone_parsed_spec()
invoked at this recursive call should takes care of appending the list of
table references encountered in the specification of this CTE cte1 to the
list of table references created for the query. And it should do it after
the call of LEX::resolve_references_to_cte() that resolves references to
CTEs defined outside of the specification of cte1 because this call may
invoke the parser again for specifications of other CTEs and  the table
references from their specifications must ultimately appear in the global
list of table references of the query.
  The code of With_element::clone_parsed_spec() misplaced the call of
LEX::resolve_references_to_cte(). As a result LEX::query_tables_last used
for the query that was supposed to point to the field 'next_global' of the
last element in the global list of table references actually pointed to
'next_global' of the previous element.
  The above inconsistency certainly caused serious problems when table
references used in the stored functions invoked in cloned specifications
of CTEs were added to the global list of table references.
2021-11-16 18:56:43 -08:00
Marko Mäkelä
6841d1afdd Merge 10.5 into 10.6 2021-11-16 17:15:13 +02:00
Marko Mäkelä
ebb15f986f MDEV-27059 page_zip_dir_insert() may corrupt ROW_FORMAT=COMPRESSED tables
In commit 7ae21b18a6 (MDEV-12353)
the recovery of ROW_FORMAT=COMPRESSED tables was changed.
Changes would be logged in a physical format for the compressed
page image, so that the page need not be decompressed or compressed
during recovery.

page_zip_write_rec(): Log any update of the delete-mark flag in the
ROW_FORMAT=COMPRESSED page.

page_zip_dir_insert(): Copy the delete-mark flag. A delete-marked
record may be inserted by btr_cur_pessimistic_update() via
btr_cur_insert_if_possible(), page_cur_tuple_insert(),
page_cur_insert_rec_zip(). In the observed scenario, it was
an ROLLBACK. Presumably, the test case involved repeated DELETE
and INSERT of the same key, or updating a key back and forth.
This change alone might make the adjustment in page_zip_write_rec()
redundant, but we play it safe because we failed to create a
minimal test case for this scenario.
2021-11-16 17:13:15 +02:00
Marko Mäkelä
89ab2538c5 MDEV-27028 [ERROR] [FATAL] InnoDB: SYS_VIRTUAL.TABLE_ID mismatch
If the server is killed during any DDL operation that is about to
delete an .ibd file, recovery could crash when attempting to load
the table definition of the being-dropped table. By design of
commit 1bd681c8b3 (MDEV-25506 part 3),
a table whose name starts with #sql-ib in the data dictionary may
belong to an uncommitted transaction. So, we must ignore any missing
SYS_COLUMNS, SYS_FIELDS, and SYS_VIRTUAL records for such tables.

The "ID mismatch" error messages were misleading; they really mean
"record not found".
2021-11-16 16:31:45 +02:00
Marko Mäkelä
dc8def73f7 Merge 10.5 into 10.6 2021-11-16 16:30:45 +02:00
Marko Mäkelä
09205a1c9a Merge 10.4 into 10.5 2021-11-16 14:26:13 +02:00
Marko Mäkelä
0269d491ea MDEV-27047: Replication fails to remove affected queries from query cache
Rows_log_event::do_apply_event(): Correct the mistake that was made in
the merge 5f8561a6bc.

In Galera, the query cache will be invalidated near the end
of the function.
2021-11-16 13:58:22 +02:00
Marko Mäkelä
079516f00e MDEV-27016: Assertion 'id.page_no() < space.size' failed
buf_flush_check_neighbors(): Relax a debug assertion that
could fail for the very last page(s) of a ROW_FORMAT=COMPRESSED tables          using a 1024-byte or 2048-byte page size.

This assertion started to fail after
commit d09426f9e6 (MDEV-26537)
modified the .ibd file extension to occur in steps of 4096 bytes.
2021-11-16 12:49:51 +02:00
Vladislav Vaintroub
c5e09bf81a MDEV-27056 Windows upgrade_wizard - CloseHandle() on invalid (already closed) pipe handle 2021-11-16 01:51:35 +01:00
Vladislav Vaintroub
9e9e91b3c2 Windows build - fix signtool search path to take modern SDKs into account 2021-11-16 01:51:22 +01:00
Vladislav Vaintroub
72afeaf8a5 MDEV-18439 Windows builds should have core_file=1 by default
reapply patch 96b472c0ae
It was not merged correctly into 10.5+
2021-11-13 10:31:46 +01:00
Thirunarayanan Balathandayuthapani
d270525dfd MDEV-23805 Make Online DDL to Instant DDL when table is empty
- In ha_innobase::prepare_inplace_alter_table(), InnoDB should
check whether the table is empty. If the table is empty then
server should avoid downgrading the MDL after prepare phase.
It is more like instant alter, does change only in dicationary
and metadata.

- Changed few debug test case to make non-empty DDL table
2021-11-12 17:46:35 +05:30
Vladislav Vaintroub
e58a312e42 Merge branch '10.5' into 10.6 2021-11-12 00:33:48 +01:00
Vladislav Vaintroub
5b21a8fafc MDEV-27030 vcol.vcol_keys_myisam fails on Windows x64, with Visual Studio 2022
Upon investigation, decided this to be a compiler bug
(happens with new compiler, on code that did not change for the last 15 years)

Fixed by de-optimizing single function remove_key(), using MSVC pragma
2021-11-12 00:32:39 +01:00
Vladislav Vaintroub
c5380c30b5 Merge branch '10.3' into 10.4 2021-11-12 00:16:37 +01:00
Vladislav Vaintroub
7ea12742d3 Merge branch '10.2' into 10.3 2021-11-12 00:08:53 +01:00
Vladislav Vaintroub
628c281db6 MDEV-27030 vcol.vcol_keys_myisam fails on Windows x64, with Visual Studio 2022
Upon investigation, decided this to be a compiler bug
(happens with new compiler, on code that did not change for the last 15 years)

Fixed by de-optimizing single function remove_key(), using MSVC pragma
2021-11-11 22:12:12 +01:00
Brandon Nesterenko
3163d9e60c MDEV-26991: CURRENT_TEST: main.mysql_binary_zero_insert 'grep' is not recognized as an internal or external command, operable program or batch file.
Removed grep from mysqldump command stream and instead,
extend the search_file pattern to search for rows containing
binary zeros instead of any occurance of '00' in the input
2021-11-11 06:19:19 -07:00
Sergei Krivonos
94ef277b5b MDEV-23766: fix by assert (Windows) 2021-11-10 20:53:24 +02:00
Thirunarayanan Balathandayuthapani
3480c3f95b MDEV-26121 [Note] InnoDB: Resetting invalid page
In dict_index_t::clear(), InnoDB frees all the page except root page.
root page leaf segment has reset and does reinitialize again.
t in fseg_create(), we do have the assumption that only
FIL_PAGE_TYPE_TRX_SYS or FIL_PAGE_TYPE_TRX_SYS page should
be re-created for non-full-crc32 format. This assumption is wrong
in case of rollback of bulk insert operation.
2021-11-10 11:35:19 +05:30
Sergei Petrunia
04ad98b500 MDEV-23766: Re-add Json_writer unit test. 2021-11-09 17:07:23 +02:00
Vladislav Vaintroub
c1e1ca20f4 In case WITH_WSREP is enabled, build wsrep as plugin
If it is not enabled, build wsrep as static "stub" library from wsrep_dummy.cc
´

Allow static plugins to export symbols (on Unix)
wsrep_info relies on wsrep defined symbols (e.g  LOCK_wsrep_config_state)
exported from mysqld
2021-11-09 17:04:49 +02:00
Vladislav Vaintroub
009f3e06f3 improve build, allow sql library to be built in parallel with builtins 2021-11-09 17:02:45 +02:00
Vladislav Vaintroub
7acf31783c Add new option NOT_EMBEDDED, for plugins
Means, plugin will not be available in embedded, even if compiled-in
2021-11-09 16:57:27 +02:00
Sergei Krivonos
f7c6c02a06 Revert "improve build, allow sql library to be built in parallel with builtins"
This reverts commit 1a3570dec3.
2021-11-09 15:44:07 +02:00
Sergei Krivonos
ff08e948d3 Revert "In case WITH_WSREP is enabled, build wsrep as plugin"
This reverts commit e45f7f485a.
2021-11-09 15:44:07 +02:00
Sergei Krivonos
9960b98267 MDEV-23766: buildfix: postpone new unittest until its dependency resolution 2021-11-09 15:44:07 +02:00
Thirunarayanan Balathandayuthapani
3989d3800d MDEV-27006 Assertion `!lock_trx_has_sys_table_locks(trx)' failed in dberr_t row_discard_tablespace_for_mysql(dict_table_t*, trx_t*)
- InnoDB import operation fails when it tries to unlock data
dictionary lock before releasing the lock on system tables.
2021-11-09 18:08:12 +05:30
Sergei Krivonos
e9b76b896a MDEV-23766: fix by my_json_writer test 2021-11-09 12:06:49 +02:00
Sergei Petrunia
5e988ff80f MDEV-23766: Make Json_writer assert when one tries to author invalid JSON
- Add unit test.
2021-11-09 12:06:49 +02:00
Sergei Petrunia
c9b5b9321f MDEV-23766: Make Json_writer assert when one tries to author invalid JSON
Code cleanup: Remove Json_writer::is_on_fmt_helper_call. We already
maintain this state in fmt_helper.
2021-11-09 12:06:49 +02:00
Vladislav Vaintroub
e45f7f485a In case WITH_WSREP is enabled, build wsrep as plugin
If it is not enabled, build wsrep as static "stub" library from wsrep_dummy.cc
´

Allow static plugins to export symbols (on Unix)
wsrep_info relies on wsrep defined symbols (e.g  LOCK_wsrep_config_state)
exported from mysqld
2021-11-09 12:06:49 +02:00
Vladislav Vaintroub
1a3570dec3 improve build, allow sql library to be built in parallel with builtins 2021-11-09 12:06:49 +02:00
Sergei Krivonos
b17576322b MDEV-23766: add Json_writer consistency asserts to check array/object sequence 2021-11-09 12:06:49 +02:00
Sergei Krivonos
cf047efd42 MDEV-23766: Fix get_best_disjunct_quick by assert:
in test main.range_vs_index_merge one path requires named JSON object:
assert.c:0(.annobin_assert.c_end)[0x7fe9d2270a76]
sql/my_json_writer.cc:43(Json_writer::on_start_object())[0x555e284f048a]
sql/my_json_writer.cc:59(Json_writer::start_object())[0x555e284ee6e8]
sql/my_json_writer.h:377(Json_writer_object::Json_writer_object(THD*))[0x555e281dce11]
sql/opt_range.cc:5137(get_best_disjunct_quick(PARAM*, SEL_IMERGE*, double))[0x555e287c576b]
sql/opt_range.cc:5492(merge_same_index_scans(PARAM*, SEL_IMERGE*, TRP_INDEX_MERGE*, double))[0x555e287c6cf6]
sql/opt_range.cc:5287(get_best_disjunct_quick(PARAM*, SEL_IMERGE*, double))[0x555e287c607a]
sql/opt_range.cc:3000(SQL_SELECT::test_quick_select

another one requires unnamed JSON:
mariadbd: /home/name/server/sql/my_json_writer.cc:379: bool Single_line_formatting_helper::on_add_member(const char*, size_t): Assertion `state== INACTIVE || state ==
assert.c:0(.annobin_assert.c_end)[0x7f33d8df8a76]
sql/my_json_writer.cc:380(Single_line_formatting_helper::on_add_member(char const*, unsigned long))[0x558362f6a717]
sql/my_json_writer.cc:150(Json_writer::add_member(char const*, unsigned long))[0x558362f69a91]
sql/my_json_writer.cc:146(Json_writer::add_member(char const*))[0x558362f69a5f]
sql/my_json_writer.h:383(Json_writer_object::Json_writer_object(THD*, char const*))[0x558362ceccaa]
sql/opt_range.cc:5139(get_best_disjunct_quick(PARAM*, SEL_IMERGE*, double))[0x5583632407d0]
sql/opt_range.cc:3000(SQL_SELECT::test_quick_select
2021-11-09 12:06:49 +02:00
Sergei Krivonos
1f1ee085fb MDEV-23766: Fix fix_semijoin_strategies_for_picked_join_order by assert 2021-11-09 12:06:49 +02:00
Marko Mäkelä
ba4f8e317d MDEV-26826 fixup for Valgrind and MemorySanitizer
The debug assertion that was added in
commit 9b967c4c31
tripped Valgrind and MemorySanitizer.

buf_block_init(): Assert that block->page.hash was zero-initialized.
2021-11-09 09:38:12 +02:00
Marko Mäkelä
25ac047baf Merge 10.5 into 10.6 2021-11-09 09:11:50 +02:00