Commit graph

195032 commits

Author SHA1 Message Date
Daniel Black
8b92e346b1 Merge 10.6 into 10.7 2022-03-25 14:31:59 +11:00
Daniel Black
63f76d3b98 Deb: enable pmem on riscv64 2022-03-25 12:01:51 +11:00
Daniel Black
1b2ee693b7 MDEV-28153: Debian autobake- use absolute dependencies rather than a buildtime detection
Fixing the version of debian/ubuntu dependencies in 10.6
removes the apt-cache checking of libpmem and liburing
dependencies.

By arranging the checks earliest to latest, we unconditionally
change the dependences in earlier versions, and in later versions
we perform architecture checks to see if there is a dependency
on this architecture before removing/changing.

This takes from the architecture information on Ubuntu[1,2]
and Debian[3,4].

[1] https://packages.ubuntu.com/search?suite=all&arch=any&searchon=names&keywords=liburing-dev
[2] https://packages.ubuntu.com/search?suite=all&arch=any&searchon=names&keywords=libpmem-dev
[3] https://packages.debian.org/search?suite=all&section=all&arch=any&searchon=names&keywords=liburing-dev
[4] https://packages.debian.org/search?suite=all&section=all&arch=any&searchon=names&keywords=libpmem-dev
2022-03-25 12:01:51 +11:00
Daniel Black
ec62f46a61 Merge 10.5 to 10.6 2022-03-25 11:31:49 +11:00
Daniel Black
157a838b19 MDEV-28153: Debian autobake- use absolute dependencies rather than a buildtime detection
While moving to a prescribed dependencies in MDEV-28011, an error was made
in the merge. The Ubuntu and Debian supported architectures of rocksdb-tools
are different and need to be treated as such.

This actually had no effect as our support of mariadb-plugin-rocksdb was never
different to the distro support of rocksdb-tools. Some notes where added
to this affect.

There is also nothing to do for Debian sid, and never should be.

The differentiation and grouping of distro codenames is for convenience in
merging upwards as more dependencies change across distro versions.

The fixing of versions rather than relying on apt-cache to be correct prevents
unstable changes between releases, and potentially uninstallable packages like
happened in MDEV-28014.

Correct comment about zstd to MDEV-16525
2022-03-25 11:01:09 +11:00
Rucha Deodhar
0da5f45a50 MDEV-27819: func_2.xxx_charset skipped after renaming utf8 to utf8mb3
Analysis: utf8 character set is now utf8mb3 or utf8mb4. So charset_master
is not able to find utf8 at the beginning of test. Hence it skips the tests
that use charset_master.
Fix: rename utf8 to utf8mb3 in charset_master.
2022-03-24 20:30:04 +05:30
Marko Mäkelä
8684af76e3 MDEV-28137 Some memory transactions are unnecessarily complex
buf_page_get_zip(): Do not perform a system call inside a
memory transaction. Instead, if the page latch is unavailable,
abort the memory transaction and let the fall-back code path
wait for the page latch.

buf_pool_t::watch_remove(): Return the previous state of the block.

buf_page_init_for_read(): Use regular stores for moving the
buffer fix count of watch_remove() to the new block descriptor.

A more extensive version of this was reviewed by Daniel Black
and tested with Intel TSX-NI by Axel Schwenke and Matthias Leich.
My assumption that regular loads and stores would execute faster
in a memory transaction than operations like std::atomic::fetch_add()
turned out to be incorrect.
2022-03-24 16:09:04 +02:00
Marko Mäkelä
b101f19d29 MDEV-23974 fixup: rpl.rpl_gtid_stop_start fails
The call mtr.add_suppression() that was added
in commit 75b7cd680b
for MemorySanitizer and Valgrind runs is causing
a result difference for the test rpl.rpl_gtid_stop_start.
Let us disable the binlog for executing that statement.

Also, the test perfschema.statement_program_lost_inst
would fail due to the changes to have_innodb.inc in this commit.
To compensate for that, we will make more --suite=perfschema
tests run without InnoDB, and explicitly enable InnoDB in
those tests that depend on a transactional storage engine.
2022-03-24 13:43:58 +02:00
Daniel Black
0444d86baf Merge 10.7 2022-03-24 22:01:27 +11:00
Daniel Black
e86986a157 Merge 10.6 into 10.7 2022-03-24 18:57:07 +11:00
Sergei Golubchik
d2c019b200 MDEV-28107 S3 doesn't build if CURL is in non-default location 2022-03-23 17:42:15 +01:00
Sergei Golubchik
2ca3861b55 MDEV-28106 S3 tries to include thread.h while compiling on Windows
S3 engine and libmarias3 don't compile on Windows

also fixes:
MDEV-28104 Typo in storage/maria/s3_func.c: 'FN_DECVHAR
2022-03-23 17:42:15 +01:00
Marko Mäkelä
75b7cd680b MDEV-23974 Tests fail due to [Warning] InnoDB: Trying to delete tablespace
A few regression tests invoke heavy flushing of the buffer pool
and may trigger warnings that tablespaces could not be deleted
because of pending writes. Those warnings are to be expected
during the execution of such tests.

The warnings are also frequently seen with Valgrind or MemorySanitizer.
For those, the global suppression in have_innodb.inc does the trick.
2022-03-23 16:42:43 +02:00
Marko Mäkelä
44231dc6d5 Cleanup: have_sanitizer='ASAN,UBSAN'
This was suggested by Sergei Golubhick.
Fixes up commit b91a123d8c
2022-03-23 16:41:58 +02:00
Thirunarayanan Balathandayuthapani
c3ca729ba8 MDEV-27858 Assertion `page_dir_get_n_heap(new_page) == 2U' failed in PageBulk::init
- InnoDB should check whether bulk transaction id set to its own
transaction id before start bulk insert operation.

- When bulk insert failure happens, InnoDB should set the error info
of the transaction.
2022-03-23 10:15:52 +05:30
Tuukka Pasanen
9c57bbdad0 MDEV-28120 Remove false-positive Lintian linking error
Lintian uses objdump-command to determine is share object linked against
C-libary. Compression provider shared objects does not have any symbols
from C-library they only depend compression library and MariaDB
libaries. This is false-positive as Lintian error itself is correct as
shared object does not share any symbols from for example Glibc.

Added overrides suppress Lintian errors:
 * E: mariadb-plugin-provider-bzip2: library-not-linked-against-libc usr/lib/mysql/plugin/provider_bzip2.so
 * E: mariadb-plugin-provider-lz4: library-not-linked-against-libc usr/lib/mysql/plugin/provider_lz4.so
 * E: mariadb-plugin-provider-lzma: library-not-linked-against-libc usr/lib/mysql/plugin/provider_lzma.so
 * E: mariadb-plugin-provider-lzo: library-not-linked-against-libc usr/lib/mysql/plugin/provider_lzo.so
 * E: mariadb-plugin-provider-snappy: library-not-linked-against-libc usr/lib/mysql/plugin/provider_snappy.so
2022-03-23 13:49:24 +11:00
Tuukka Pasanen
770f62bf3e MDEV-28135: Add missing conflict and breaks for MariaDB 10.6
There is missing conflict and breaks for MariaDB 10.6 which
can cause upgrading from 10.6 not be clean and end up with
errors on Debian.
2022-03-23 13:34:07 +11:00
Tuukka Pasanen
1960f7b224 MDEV-27033: Clean lintian 'extended-description-is-empty' errors
Fix Debian lintian errors:
  E: mariadb-plugin-provider-bzip2: extended-description-is-empty
  E: mariadb-plugin-provider-lz4: extended-description-is-empty
  E: mariadb-plugin-provider-lzma: extended-description-is-empty
  E: mariadb-plugin-provider-lzo: extended-description-is-empty
  E: mariadb-plugin-provider-snappy: extended-description-is-empty
by adding Debian Policy Manual section 3.4 (The description
of a package) extended descriptions to mariadb-plugin-provide-*
packages
2022-03-23 13:14:00 +11:00
Andrei
5ccd845d51 MDEV-27760 event may non stop replicate in circular semisync setup
MDEV-21117 had to relax own events acceptance condition for a case
when a former semisync master server recovers after crash as the
semisync slave. That however admitted a possibility for endless event
"orbiting" in the non-strict slave gtid mode of semisync circular
setup.

The same server-id event termination is restored now for
the non-strict gtid mode to follow regular rules (that is it's ignored
unless @@global.replicate_same_server_id allows it in).

To address MDEV-21117 recovery agenda,
in the strict gtid mode and the transaction's gtid ordered strictly
greater than the current slave gtid state, the same server-id
transaction is accepted.

The gtid strict mode is safe to accept transactions even if
the slave state were not set correct by the user, e.g
at the former master.
An added test shows a typical out-of-order error at execution so
no data corruption is guaranteed in such a case.
2022-03-22 19:20:19 +02:00
Julius Goryavsky
35725df6e2 MDEV-27524 addendum: fix for bug introduced by automatic migration 2022-03-22 03:23:32 +01:00
Marko Mäkelä
8840583a92 MDEV-27909 InnoDB: Failing assertion: state == TRX_STATE_NOT_STARTED ... on DDL
The fix in commit 6e390a62ba (MDEV-26772)
was a step to the right direction, but implemented incorrectly.
When an InnoDB persistent statistics table cannot be locked immediately,
we must not let row_mysql_handle_errors() to roll back the transaction.

lock_table_for_trx(): Add the parameter no_wait (default false)
for an immediate return of DB_LOCK_WAIT in case of a conflict.

ha_innobase::delete_table(), ha_innobase::rename_table():
Pass no_wait=true to lock_table_for_trx() when needed,
instead of temporarily setting THDVAR(thd, lock_wait_timeout) to 0.
2022-03-18 10:52:08 +02:00
Otto Kekäläinen
9595ea8992 Deb: Sync Salsa-CI from Debian MariaDB 10.5 repository
Since Debian Sid now has MariaDB 10.6, we can't do any upgrade tests in
Debian Sid for the 10.5 branch anymore. It would just fail with downgrade
errors.

Also, since MariaDB 10.5 is no longer in Sid, we can't even test 10.5.x
to 10.5.y upgrades in Sid.

Instead the 10.5 branch salsa-ci.yml should run all builds and tests based
on Debian Bullseye, which has MariaDB 10.5 (only).

To achieve this, essentially sync most the the salsa-ci.yml contents from
https://salsa.debian.org/mariadb-team/mariadb-10.5/-/tree/bullseye

Also add a couple Lintian overrides to make Salsa-CI pass.

NOTE TO MERGERS: This commit is intended for the 10.5 branch only, do not
merge anything from it on 10.6 or any other branch.
2022-03-17 20:41:50 -07:00
Daniel Black
065f995e6d Merge branch 10.5 into 10.6 2022-03-18 12:17:11 +11:00
Sergei Golubchik
bf8dc0be9e fix columnstore compilation after 33c30da165
normally, one has to include my_global.h before
including psi/psi*.h files. ColumnStore cannot do it,
so it needs a workaround.
2022-03-17 12:13:19 +01:00
Marko Mäkelä
06e3bc4390 MDEV-17841 fixup: GCC -Wmaybe-uninitialized
In commit ab38b7511b
an added "goto err" would seemingly cause a read of
an uninitialized variable old_info if errpos>=5.

However, because we would have errpos=0 at that point,
there was no real error.
2022-03-17 10:33:06 +02:00
Daniel Black
b73d852779 Merge 10.4 to 10.5 2022-03-17 17:03:24 +11:00
Marko Mäkelä
ee80c19633 MDEV-26551 InnoDB crash on multiple concurrent SHOW TABLE STATUS
dict_get_and_save_data_dir_path(): Protect the operation with
dict_table_t::lock_mutex and avoid unnecessary memory allocation.
2022-03-16 17:19:13 +02:00
Thirunarayanan Balathandayuthapani
31ad9277fe MDEV-28079 Shutdown hangs after altering innodb partition fts table
- InnoDB purge waits at resume_FTS() while shutting down.
This happens after altering the FTS innodb partition table.
stop_FTS() has been called for each partition, but it calls
resume_FTS() only once and it leads to hang during shutdown.
This issue was introduced by
commit 1bd681c8b3c5213ce1f7976940a7dc38b48a0d39(MDEV-25506).
2022-03-16 19:20:27 +05:30
Daniel Black
069139a549 Merge 10.3 to 10.4
extra2_read_len resolved by keeping the implementation
in sql/table.cc by exposed it for use by ha_partition.cc

Remove identical implementation in unireg.h
(ref: bfed2c7d57)
2022-03-16 16:39:10 +11:00
Daniel Black
6a2d88c132 Merge 10.2 to 10.3 2022-03-16 12:51:22 +11:00
Alexander Barkov
0e63023cb8 Merge branch 10.2 into 10.3 2022-03-16 12:49:13 +11:00
Daniel Black
b2c81e06b0 MDEV-27955 main.func_json_notembedded test fails on out-of-memory
Uses 500M+ of memory by repeating an 8 byte sequence 62.5M times.

Reduce the number of repeats on string reduced by 100 times.

Tested by applying against the reverted MDEV-24909 code. 1000 times
reduction was too much, but 100 still managed to trigger the bug.
2022-03-16 09:41:54 +11:00
Daniel Black
57dbe8785d MDEV-23915 ER_KILL_DENIED_ERROR not passed a thread id (part 2)
Per Marko's comment in JIRA, sql_kill is passing the thread id
as long long. We change the format of the error messages to match,
and cast the thread id to long long in sql_kill_user.
2022-03-16 09:37:45 +11:00
Daniel Black
99837c61a6 MDEV-23915 ER_KILL_DENIED_ERROR not passed a thread id
The 10.5 test error main.grant_kill showed up a incorrect
thread id on a big endian architecture.

The cause of this is the sql_kill_user function assumed the
error was ER_OUT_OF_RESOURCES, when the the actual error was
ER_KILL_DENIED_ERROR. ER_KILL_DENIED_ERROR as an error message
requires a thread id to be passed as unsigned long, however a
user/host was passed.

ER_OUT_OF_RESOURCES doesn't even take a user/host, despite
the optimistic comment. We remove this being passed as an
argument to the function so that when MDEV-21978 is implemented
one less compiler format warning is generated (which would
have caught this error sooner).

Thanks Otto for reporting and Marko for analysis.
2022-03-16 09:37:45 +11:00
Marko Mäkelä
dc4b7f382b Merge 10.6 into 10.7 2022-03-15 15:25:31 +02:00
Marko Mäkelä
4ef44cc2f9 Merge 10.5 into 10.6 2022-03-15 14:49:24 +02:00
Marko Mäkelä
73fee39ea6 MDEV-27985 buf_flush_freed_pages() causes InnoDB to hang
buf_flush_freed_pages(): Assert that neither buf_pool.mutex
nor buf_pool.flush_list_mutex are held. Simplify the loops.
Return the tablespace and the number of pages written or punched.

buf_flush_LRU_list_batch(), buf_do_flush_list_batch():
Release buf_pool.mutex before invoking buf_flush_space().

buf_flush_list_space(): Acquire the mutexes only after invoking
buf_flush_freed_pages().

Reviewed by: Thirunarayanan Balathandayuthapani
2022-03-15 14:44:22 +02:00
Marko Mäkelä
de4ec44b4f Fix clang -Wtypedef-redefinition
Include my_global.h before mysql.h so that the latter will not
redefine my_socket and my_ulonglong.

Fixup for commit 77c184df7c
2022-03-15 12:57:39 +02:00
Marko Mäkelä
00896db1c5 MDEV-25214 Crash in fil_space_t::try_to_close
fil_space_t::try_to_close(): Tolerate a tablespace that has no
data files attached. The function fil_ibd_create() initially
creates and attaches a tablespace with no files, and invokes
fil_space_t::add() later.

fil_node_open_file(): After releasing and reacquiring fil_system.mutex,
check if the file was already opened by another thread. This avoids
an assertion failure !node->is_open() in fil_node_open_file_low().

These failures were reproduced with the test
innodb.table_definition_cache_debug and the fix of MDEV-27985.
2022-03-15 10:37:13 +02:00
Marko Mäkelä
e1246775a9 Merge 10.4 into 10.5 2022-03-15 08:32:28 +02:00
Marko Mäkelä
9c6135e81f Merge 10.3 into 10.4 2022-03-15 08:10:35 +02:00
Daniel Black
a950086036 Merge 10.2 (part) into 10.3
commit '6de482a6fefac0c21daf33ed465644151cdf879f'

10.3 no longer errors in truncate_notembedded.test
but per comments, a non-crash is all that we are after.
2022-03-15 16:44:52 +11:00
Hugo Wen
dafc5fb9c1 MDEV-27342: Fix issue of recovery failure using new server id
Commit 6c39eaeb1 made the crash recovery dependent on server_id.
The crash recovery could fail when restoring a new instance from
original crashed data directory USING A NEW SERVER ID.

The issue doesn't exist in previous major versions before 10.6.

Root cause is when generating the input XID to be searched in the hash,
server id is populated with the current server id.
So if the server id changed when recovering, the XID couldn't be found
in the hash due to server id doesn't match.

This fix is to use original server id when creating the input XID
object in function `xarecover_do_commit_or_rollback`.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
2022-03-14 19:57:10 -07:00
Thirunarayanan Balathandayuthapani
1c43660aea MDEV-28060 Online DDL fails while checking for instant alter condition
- InnoDB fails to skip newly created column while checking for
change column when table is in redundant row format. This issue
is caused the MDEV-18035 (ccb1acbd3c)
2022-03-14 22:35:11 +05:30
Alexander Barkov
03c3dc6365 MDEV-23210 Assertion `(length % 4) == 0' failed in my_lengthsp_utf32 on ALTER TABLE, SELECT and INSERT
Problem:
Parse-time conversion from binary to tricky character sets like utf32
produced ill-formed strings. So, later a chash happened in debug builds,
or a wrong SHOW CREATE TABLE was returned in release builds.

Fix:

1. Backporting a few methods from 10.3:
  - THD::check_string_for_wellformedness()
  - THD::convert_string() overloads
  - THD::make_text_string_connection()

2. Adding a new method THD::reinterpret_string_from_binary(),
   which makes sure to either returns a well-formed string
   (optionally prepending with zero bytes), or returns an error.
2022-03-14 14:42:59 +04:00
Marko Mäkelä
e67d46e4a1 Merge 10.6 into 10.7 2022-03-14 11:30:32 +02:00
Marko Mäkelä
572e34304e Merge 10.5 into 10.6 2022-03-14 10:59:46 +02:00
Marko Mäkelä
258c34f17c MDEV-28050: clang -Wtypedef-redefinition when PLUGIN_S3=NO
Let us remove the redundant typedef.
This problem was revealed by
commit 77c184df7c
2022-03-14 10:42:50 +02:00
Marko Mäkelä
c2146ce774 MDEV-24841: More workarounds
For some reason, the tests of the MemorySanitizer build on 10.5 failed
with both clang 13 and clang 14 with SIGSEGV. On 10.6 where it worked
better, some more places to work around were identified.
2022-03-14 10:37:39 +02:00
Sergei Golubchik
f217c76189 mtr: fix --source lines detection
mysqltest allows leading spaces before `--`, so mtr should too
2022-03-14 08:55:59 +01:00