Commit graph

26778 commits

Author SHA1 Message Date
Sergei Golubchik
d77aaa6994 MCOL-5437 columnstore fails to compile to due old cs->casedn_multiply use
update columnstore
2023-03-10 09:59:22 +01:00
Marko Mäkelä
c5fdb988b7 Merge 10.11 into 11.0 2023-03-06 16:06:52 +02:00
Marko Mäkelä
9267160c11 Merge 10.10 into 10.11 2023-03-06 13:39:12 +02:00
Marko Mäkelä
4ccb2be65f Merge 10.9 into 10.10 2023-03-06 13:38:39 +02:00
Marko Mäkelä
46a7603813 Merge 10.8 into 10.9 2023-03-06 13:38:16 +02:00
Marko Mäkelä
669a0c6efb Merge 10.6 into 10.8 2023-03-06 13:37:12 +02:00
Marko Mäkelä
077425a659 MDEV-30311 system-wide max transaction id corrupted after changing the undo tablespaces
This fixes a regression due to MDEV-19229. InnoDB would fail to maintain
the maximum transaction ID when it changes and reinitializes the number
of undo tablespaces. InnoDB should maintain the maximum transaction ID
in TRX_RSEG_MAX_TRX_ID of system rollback segment header.

srv_undo_tablespaces_reinit(): Preserve the system-wide maximum
transaction identifier in the TRX_RSEG_MAX_TRX_ID field of
the first rollback segment. If needed, upgrade the page to the
MariaDB 10.3 format first. All this must be done in the same
atomic mini-transaction that will reinitialize the TRX_SYS page.

Before MariaDB Server 10.3, InnoDB persisted the maximum transaction
identifier only in the TRX_SYS page. MariaDB 10.3 started to treat that
page as a read-only directory of rollback segments, and the maximum
transaction identifier will be recovered from TRX_RSEG_MAX_TRX_ID or
from undo logs. Since a change of innodb_undo_tablespaces is only
allowed when no undo log records exist, the only place to store the
persistent maximum transaction identifier is in TRX_RSEG_MAX_TRX_ID
of one of the rollback segment header pages.

The bug was observed when the database was upgraded directly from MySQL 5.7
or earlier, or from MariaDB Server 10.2 or earlier, to multiple
innodb_undo_tablespaces. On a restart of MariaDB after the upgrade,
the transaction identifier would be reported to be smaller than during
the upgrade:

2023-03-03 10:43:57 0 [Note] InnoDB: log sequence number 2762352; transaction id 1794
2023-03-03 10:44:17 0 [Note] InnoDB: log sequence number 2786076; transaction id 770
2023-03-03 11:38:43 +02:00
Monty
bc3596fe12 MMAP does not work reliable on windows
This error was discovered while working on
MDEV-30540 Wrong result with IN list length reaching
           IN_PREDICATE_CONVERSION_THRESHOLD

Failing test: cte_recursive.test

If one writes to a file, then truncates it and then call mmap() over the
file_size + 7, then the file size changes to 7. (On Linux mmap() does not
change file size).
This caused _ma_read_rnd_dynamic_record() to believe that there are more
records in the data file, which is not the case, and the table will be
marked as corrupted.
Fixed by disabling mmap() in Aria on Windows.
2023-03-02 13:11:54 +02:00
Monty
37edbbf2d3 Don't log delete_all_rows() for temporary Aria files to transaction log
- This was just a small performance issue, not a crashing bug.
2023-03-02 13:11:54 +02:00
Monty
bf9aa8687f Fixes to make dbug traces from Windows easier to compare with Unix traces
- Remove DBUG calls from my_winfile.c where call and parameters
  are already printed by mysys.
- Remove DBUG from my_get_osfhandle() and my_get_open_flags() to remove
  DBUG noise.
- Updated convert-debug-for-diff to take into account windows.
- Changed some DBUG_RETURN(function()) to tmp=function(); DBUG_RETURN(tmp);
  This is needed as Visual C++ prints for DBUG binaries a trace for
  func_a()
  {
    DBUG_ENTER("func_a");
    DBUG_RETURN(func_b())
  }
  as
  >func_a
  <func_a
  >func_b
  <func_b
instead of when using gcc:
  >func_a
  | >func_b
  | <func_b
  <func_a
2023-03-02 13:11:54 +02:00
Thirunarayanan Balathandayuthapani
49e2b50d59 MDEV-30341 Reset check_foreigns, check_unique_secondary variables
- InnoDB fails to reset the check_foreigns and check_unique_secondary
in trx_t::free(), trx_t::commit_cleanup(). This lead to bulk insert
in internal innodb fts table operation.
2023-03-02 15:49:21 +05:30
Thirunarayanan Balathandayuthapani
550b8d76b3 MDEV-30752 Assertion `!index->is_ibuf()' failed around
cmp_dtuple_rec_with_match_bytes

- InnoDB shouldn't use the adaptive hash index for change buffer indexes.
2023-03-02 11:22:40 +05:30
Marko Mäkelä
085d0ac238 Merge 10.5 into 10.6 2023-02-28 16:05:21 +02:00
Marko Mäkelä
c14a39431b MDEV-30753 Possible corruption due to trx_purge_free_segment()
Starting with commit 0de3be8cfd (MDEV-30671),
the field TRX_UNDO_NEEDS_PURGE lost its previous meaning.
The following scenario is possible:

(1) InnoDB is killed at a point of time corresponding to the durable
execution of some fseg_free_step_not_header() but not
trx_purge_remove_log_hdr().
(2) After restart, the affected pages are allocated for something else.
(3) Purge will attempt to access the newly reallocated pages when looking
for some old undo log records.

trx_purge_free_segment(): Invoke trx_purge_remove_log_hdr() as the first
thing, to be safe. If the server is killed, some pages will never be
freed. That is the lesser evil. Also, before each mtr.start(), invoke
log_free_check() to prevent ib_logfile0 overrun.
2023-02-28 15:39:23 +02:00
Marko Mäkelä
a75cd0a734 MDEV-30671 follow-up: Remove the field TRX_UNDO_NEEDS_PURGE
Because downgrades from 11.0 to older MariaDB server are not possible
due to the removal of the InnoDB change buffer, there is no need to
access the field TRX_UNDO_NEEDS_PURGE anymore.
2023-02-28 13:21:31 +02:00
Marko Mäkelä
7a834d6248 Merge 10.11 into 11.0 2023-02-28 13:14:08 +02:00
Marko Mäkelä
95d51369c9 Merge 10.10 into 10.11 2023-02-28 10:52:42 +02:00
Marko Mäkelä
f14d9fa09a Merge 10.9 into 10.10 2023-02-28 10:43:29 +02:00
Marko Mäkelä
c3246e4bf0 Merge 10.8 into 10.9 2023-02-28 10:37:11 +02:00
Marko Mäkelä
6ac44ac3ab Merge 10.6 into 10.8 2023-02-28 10:36:17 +02:00
Monty
57c526ffb8 Added detection of memory overwrite with multi_malloc
This patch also fixes some bugs detected by valgrind after this
patch:

- Not enough copy_func elements was allocated by Create_tmp_table() which
  causes an memory overwrite in Create_tmp_table::add_fields()
  I added an ASSERT() to be able to detect this also without valgrind.
  The bug was that TMP_TABLE_PARAM::copy_fields was not correctly set
  when calling create_tmp_table().
- Aria::empty_bits is not allocated if there is no varchar/char/blob
  fields in the table.  Fixed code to take this into account.
  This cannot cause any issues as this is just a memory access
  into other Aria memory and the content of the memory would not be used.
- Aria::last_key_buff was not allocated big enough. This may have caused
  issues with rtrees and ma_extra(HA_EXTRA_REMEMBER_POS) as they
  would use the same memory area.
- Aria and MyISAM didn't take extended key parts into account, which
  caused problems when copying rec_per_key from engine to sql level.
- Mark asan builds with 'asan' in version strihng to detect these in
  not_valgrind_build.inc.
  This is needed to not have main.sp-no-valgrind fail with asan.
2023-02-27 19:25:44 +02:00
Marko Mäkelä
3e2ad0e918 Merge 10.5 into 10.6 2023-02-27 13:17:35 +02:00
Marko Mäkelä
0de3be8cfd MDEV-30671 InnoDB undo log truncation fails to wait for purge of history
It is not safe to invoke trx_purge_free_segment() or execute
innodb_undo_log_truncate=ON before all undo log records in
the rollback segment has been processed.

A prominent failure that would occur due to premature freeing of
undo log pages is that trx_undo_get_undo_rec() would crash when
trying to copy an undo log record to fetch the previous version
of a record.

If trx_undo_get_undo_rec() was not invoked in the unlucky time frame,
then the symptom would be that some committed transaction history is
never removed. This would be detected by CHECK TABLE...EXTENDED that
was impleented in commit ab0190101b.
Such a garbage collection leak should be possible even when using
innodb_undo_log_truncate=OFF, just involving trx_purge_free_segment().

trx_rseg_t::needs_purge: Change the type from Boolean to a transaction
identifier, noting the most recent non-purged transaction, or 0 if
everything has been purged. On transaction start, we initialize this
to 1 more than the transaction start ID. On recovery, the field may be
adjusted to the transaction end ID (TRX_UNDO_TRX_NO) if it is larger.

The field TRX_UNDO_NEEDS_PURGE becomes write-only; only some debug
assertions that would validate the value. The field reflects the old
inaccurate Boolean field trx_rseg_t::needs_purge.

trx_undo_mem_create_at_db_start(), trx_undo_lists_init(),
trx_rseg_mem_restore(): Remove the parameter max_trx_id.
Instead, store the maximum in trx_rseg_t::needs_purge,
where trx_rseg_array_init() will find it.

trx_purge_free_segment(): Contiguously hold a lock on
trx_rseg_t to prevent any concurrent allocation of undo log.

trx_purge_truncate_rseg_history(): Only invoke trx_purge_free_segment()
if the rollback segment is empty and there are no pending transactions
associated with it.

trx_purge_truncate_history(): Only proceed with innodb_undo_log_truncate=ON
if trx_rseg_t::needs_purge indicates that all history has been purged.

Tested by: Matthias Leich
2023-02-24 14:24:44 +02:00
Thirunarayanan Balathandayuthapani
db245e1140 MDEV-25984 Assertion `max_doc_id > 0' failed in fts_init_doc_id()
- rollback_inplace_alter_table() locks the fts internal tables.
At the time, insert tries to fetch the doc id from config table,
fails to lock the config table and returns doc id as 0.

fts_cmp_set_sync_doc_id(): Retry to fetch the doc id again if
it encounter DB_LOCK_WAIT_TIMEOUT error
2023-02-22 18:54:00 +05:30
Marko Mäkelä
d5d7c8ba96 MDEV-30544 Deprecate innodb_defragment and related parameters
There is a little used option innodb_defragment that would make
OPTIMIZE TABLE not rebuild the table as usual for InnoDB, but
instead cause the index B-trees to be optimized in place.

This option uses excessive locking (exclusively locking index trees).
It never covered SPATIAL INDEX or FULLTEXT INDEX. Storage space
was never reclaimed.

Because this option is not particularly useful and causes a
maintenance burden (most recently in
commit de4030e4d4),
it is best to deprecate it, to prepare for its removal.
2023-02-21 13:33:47 +02:00
Marko Mäkelä
2b13ae1a31 MDEV-29694 fixup: Remove srv_change_buffer_max_size, adjust comments 2023-02-21 12:34:20 +02:00
Vlad Lesin
a474e3278c MDEV-27701 Race on trx->lock.wait_lock between lock_rec_move() and lock_sys_t::cancel()
The initial issue was in assertion failure, which checked the equality
of lock to cancel with trx->lock.wait_lock in lock_sys_t::cancel().

If we analyze lock_sys_t::cancel() code from the perspective of
trx->lock.wait_lock racing, we won't find the error there, except the
cases when we need to reload it after the corresponding latches
acquiring.

So the fix is just to remove the assertion and reload
trx->lock.wait_lock after acquiring necessary latches.

Reviewed by: Marko Mäkelä <marko.makela@mariadb.com>
2023-02-20 20:31:24 +03:00
Alexander Barkov
7f6b648d7d MDEV-30661 UPPER() returns an empty string for U+0251 in uca1400 collations for utf8
String length growth during upper/lower conversion
in Unicode collations depends only on the underlying MY_UNICASE_INFO
used in the collation.

Maintaining a separate member CHARSET_INFO::caseup_multiply and
CHARSET_INFO::casedn_multiply duplicated this information
and caused bugs like this (when MY_UNICASE_INFO and case??_multiply
when out of sync because of incomplete CHARSET_INFO initialization).

Fix:

Changing CHARSET_INFO::caseup_multiply and CHARSET_INFO::casedn_multiply
from members to virtual functions.
The virtual functions in Unicode collations calculate case conversion
growth factors from the MY_UNICASE_INFO. This guarantees that the growth
factors are always in sync with the MY_UNICASE_INFO.
2023-02-17 17:33:27 +04:00
Thirunarayanan Balathandayuthapani
f2dc4d4c10 MDEV-30673 InnoDB recovery hangs when buf_LRU_get_free_block
is being called

- Recovery fails to release log_sys.latch for non-last batch
before preallocates the block in recv_read_in_area().
2023-02-17 18:35:13 +05:30
Marko Mäkelä
2e431ff7e6 Merge 10.11 into 11.0 2023-02-16 13:34:45 +02:00
Marko Mäkelä
1fd0099839 Merge 10.10 into 10.11 2023-02-16 11:41:18 +02:00
Marko Mäkelä
345356b868 Merge 10.9 into 10.10 2023-02-16 11:36:38 +02:00
Marko Mäkelä
0d55914d96 Merge 10.8 into 10.9 2023-02-16 10:25:34 +02:00
Marko Mäkelä
5abbe092e6 Merge 10.6 into 10.8 2023-02-16 09:17:06 +02:00
Marko Mäkelä
201cfc33e6 MDEV-30638 Deadlock between INSERT and InnoDB non-persistent statistics update
This is a partial revert of
commit 8b6a308e46 (MDEV-29883)
and a follow-up to the
merge commit 394fc71f4f (MDEV-24569).

The latching order related to any operation that accesses the allocation
metadata of an InnoDB index tree is as follows:

1. Acquire dict_index_t::lock in non-shared mode.
2. Acquire the index root page latch in non-shared mode.
3. Possibly acquire further index page latches. Unless an exclusive
dict_index_t::lock is held, this must follow the root-to-leaf,
left-to-right order.
4. Acquire a *non-shared* fil_space_t::latch.
5. Acquire latches on the allocation metadata pages.
6. Possibly allocate and write some pages, or free some pages.

btr_get_size_and_reserved(), dict_stats_update_transient_for_index(),
dict_stats_analyze_index(): Acquire an exclusive fil_space_t::latch
in order to avoid a deadlock in fseg_n_reserved_pages() in case of
concurrent access to multiple indexes sharing the same "inode page".

fseg_page_is_allocated(): Acquire an exclusive fil_space_t::latch
in order to avoid deadlocks. All callers are holding latches
on a buffer pool page, or an index, or both.
Before commit edbde4a11f (MDEV-24167)
a third mode was available that would not conflict with the shared
fil_space_t::latch acquired by ha_innobase::info_low(),
i_s_sys_tablespaces_fill_table(),
or i_s_tablespaces_encryption_fill_table().
Because those calls should be rather rare, it makes sense to use
the simple rw_lock with only shared and exclusive modes.

fil_crypt_get_page_throttle(): Avoid invoking fseg_page_is_allocated()
on an allocation bitmap page (which can never be freed), to avoid
acquiring a shared latch on top of an exclusive one.

mtr_t::s_lock_space(), MTR_MEMO_SPACE_S_LOCK: Remove.
2023-02-16 08:30:20 +02:00
Marko Mäkelä
54c0ac72e3 MDEV-30134 Assertion failed in buf_page_t::unfix() in buf_pool_t::watch_unset()
buf_pool_t::watch_set(): Always buffer-fix a block if one was found,
no matter if it is a watch sentinel or a buffer page. The type of
the block descriptor will be rechecked in buf_page_t::watch_unset().
Do not expect the caller to acquire the page hash latch. Starting with
commit bd5a6403ca it is safe to release
buf_pool.mutex before acquiring a buf_pool.page_hash latch.

buf_page_get_low(): Adjust to the changed buf_pool_t::watch_set().

This simplifies the logic and fixes a bug that was reproduced when
using debug builds and the setting innodb_change_buffering_debug=1.
2023-02-16 08:29:44 +02:00
Marko Mäkelä
9c15799462 MDEV-30397: MariaDB crash due to DB_FAIL reported for a corrupted page
buf_read_page_low(): Map the buf_page_t::read_complete() return
value DB_FAIL to DB_PAGE_CORRUPTED. The purpose of the DB_FAIL
return value is to avoid error log noise when read-ahead brings
in an unused page that is typically filled with NUL bytes.

If a synchronous read is bringing in a corrupted page where the
page frame does not contain the expected tablespace identifier and
page number, that must be treated as an attempt to read a corrupted
page. The correct error code for this is DB_PAGE_CORRUPTED.
The error code DB_FAIL is not handled by row_mysql_handle_errors().

This was missed in commit 0b47c126e3
(MDEV-13542).
2023-02-16 08:28:14 +02:00
Marko Mäkelä
cc27e5fd0e Merge 10.5 into 10.6 2023-02-16 08:17:54 +02:00
Marko Mäkelä
5300c0fb76 MDEV-30657 InnoDB: Not applying UNDO_APPEND due to corruption
This almost completely reverts
commit acd23da4c2 and
retains a safe optimization:

recv_sys_t::parse(): Remove any old redo log records for the
truncated tablespace, to free up memory earlier.
If recovery consists of multiple batches, then recv_sys_t::apply()
will must invoke recv_sys_t::trim() again to avoid wrongly
applying old log records to an already truncated undo tablespace.
2023-02-15 18:16:41 +02:00
Monty
cc182aca93 Fixed compiler warning in connect/ha_connect.cc
(fp->field_length is always >= 0)
2023-02-15 15:50:31 +03:00
Sergei Petrunia
10a974adc9 Merge 11.0-selectivity into 11.0 2023-02-15 12:03:12 +03:00
Andrew Hutchings
690fcfbd29 Fix S3 engine Coverity hits
Very minor hits found by Coverity for the S3 engine.
2023-02-14 16:27:21 +00:00
Marko Mäkelä
96a3b11d13 Merge 10.5 into 10.6 2023-02-14 15:23:23 +02:00
Thirunarayanan Balathandayuthapani
951d81d92e MDEV-30426 Assertion !rec_offs_nth_extern(offsets2, n) during bulk insert
- cmp_rec_rec_simple() fails to detect duplicate key error for
bulk insert operation
2023-02-14 15:43:33 +05:30
Thirunarayanan Balathandayuthapani
1a5c7552ea MDEV-30552 InnoDB recovery crashes when error handling scenario
- InnoDB fails to reset the after_apply variable before applying
the redo log in last batch during multi-batch recovery.
2023-02-14 14:36:17 +05:30
Thirunarayanan Balathandayuthapani
3eea2e8e10 MDEV-30551 InnoDB recovery hangs when buffer pool ran out of memory
- During non-last batch of multi-batch recovery, InnoDB holds
log_sys.mutex and preallocates the block which may intiate
page flush, which may initiate log flush, which requires
log_sys.mutex to acquire again. This leads to assert failure.
So InnoDB recovery should release log_sys.mutex before
preallocating the block.
2023-02-14 14:35:35 +05:30
Daniel Black
9656356b55 MDEV-30203 Move mysql symlinks to different package
For both Deb and RPM, create mariadb-client-compat and
mariadb-server-compat containing the mysql links to the mariadb
named executables/scripts.

The mariadb-client-core mysqlcheck was moved to mariadb-client-compat.

The symlinks in MYSQL_ADD_EXECUTABLE is tagged as a
{Client,Server}Symlinks component and placed in
the symlinks packages.

Man pages are restructured be installed into compat package
if that matches the executable.

Columnstore has a workaround as it doesn't use the cmake/plugin.cmake.

Scripts likewise have compatibility symlinks are in
the {server,client}-compat packages.

Co-author: Andrew Hutchings <andrew@linuxjedi.co.uk>

Closes 
2023-02-12 12:15:22 +01:00
Sergei Golubchik
c6f0814468 more changes to man page handling
* move them from ManPagesX component to X (works better for plugins),
  but keep ManPagesDevelopment as C/C is using it
* move backup manpages to Backup
* move plugin manpages (s3, rocksdb) to plugins
2023-02-12 12:15:22 +01:00
Marko Mäkelä
dbab3e8d90 Merge 10.6 into 10.8 2023-02-10 13:43:53 +02:00
Monty
00704aff98 Fixed bug in extended key handling when there is no primary key
Extended keys works by first checking if the engine supports extended
keys.
If yes, it extends secondary key with primary key components and mark the
secondary keys as HA_EXT_NOSAME (unique).
If we later notice that there where no primary key, the extended key
information for secondary keys in share->key_info is reset. However the
key_info->flag HA_EXT_NOSAME was not reset!

This causes some strange things to happen:
- Tables that have no primary key or secondary index that contained the
  primary key would be wrongly optimized as the secondary key could be
  thought to be unique when it was not and not unique when it was.
- The problem was not shown in EXPLAIN because of a bug in
  create_ref_for_key() that caused EQ_REF to be displayed by EXPLAIN as REF
  when extended keys where used and the secondary key contained the primary
  key.

This is fixed with:
- Removed wrong test in make_join_select() which did not detect that key
  where unique when a secondary key contains the primary.
- Moved initialization of extended keys from create_key_infos() to
  init_from_binary_frm_image() after we know if there is a usable primary
  key or not. One disadvantage with this approach is that
  key_info->key_parts may have not used slots (for keys we thought could
  be extended but could not). Fixed by adding a check for unused key_parts
  to copy_keys_from_share().

Other things:
- Simplified copying of first key part in create_key_infos().
- Added a lot of code comments in code that I had to check as part of
  finding the issue.
- Fixed some indentation.
- Replaced a couple of looks using references to pointers in C
  context where the reference does not give any benefit.
- Updated Aria and Maria to not assume the all key_info->rec_per_key
  are in one memory block (this could happen when using dervived
  tables with many keys).
- Fixed a bug where key_info->rec_per_key where not allocated
- Optimized TABLE::add_tmp_key() to only call alloc() once.
  (No logic changes)

Test case changes:
- innodb_mysql.test changed index as an index the optimizer thought
  was unique, was not. (Table had no primary key)

TODO:
- Move code that checks for partial or too long keys to the primary loop
  earlier that initally decides if we should add extended key fields.
  This is needed to ensure that HA_EXT_NOSAME is not set for partial or
  too long keys. It will also shorten the current code notable.
2023-02-10 13:35:31 +02:00