Commit graph

195664 commits

Author SHA1 Message Date
Daniel Black
63f76d3b98 Deb: enable pmem on riscv64 2022-03-25 12:01:51 +11:00
Daniel Black
1b2ee693b7 MDEV-28153: Debian autobake- use absolute dependencies rather than a buildtime detection
Fixing the version of debian/ubuntu dependencies in 10.6
removes the apt-cache checking of libpmem and liburing
dependencies.

By arranging the checks earliest to latest, we unconditionally
change the dependences in earlier versions, and in later versions
we perform architecture checks to see if there is a dependency
on this architecture before removing/changing.

This takes from the architecture information on Ubuntu[1,2]
and Debian[3,4].

[1] https://packages.ubuntu.com/search?suite=all&arch=any&searchon=names&keywords=liburing-dev
[2] https://packages.ubuntu.com/search?suite=all&arch=any&searchon=names&keywords=libpmem-dev
[3] https://packages.debian.org/search?suite=all&section=all&arch=any&searchon=names&keywords=liburing-dev
[4] https://packages.debian.org/search?suite=all&section=all&arch=any&searchon=names&keywords=libpmem-dev
2022-03-25 12:01:51 +11:00
Daniel Black
ec62f46a61 Merge 10.5 to 10.6 2022-03-25 11:31:49 +11:00
Daniel Black
157a838b19 MDEV-28153: Debian autobake- use absolute dependencies rather than a buildtime detection
While moving to a prescribed dependencies in MDEV-28011, an error was made
in the merge. The Ubuntu and Debian supported architectures of rocksdb-tools
are different and need to be treated as such.

This actually had no effect as our support of mariadb-plugin-rocksdb was never
different to the distro support of rocksdb-tools. Some notes where added
to this affect.

There is also nothing to do for Debian sid, and never should be.

The differentiation and grouping of distro codenames is for convenience in
merging upwards as more dependencies change across distro versions.

The fixing of versions rather than relying on apt-cache to be correct prevents
unstable changes between releases, and potentially uninstallable packages like
happened in MDEV-28014.

Correct comment about zstd to MDEV-16525
2022-03-25 11:01:09 +11:00
Brandon Nesterenko
cd88b0831f DBAAS-7828: Primary/replica: configuration change of autocommit=0 can not be applied
Problem:
========
When the mysql.gtid_slave_pos table uses the InnoDB engine, and
mysqld starts, it reads the table and begins a transaction. After
reading the value, it should end the transaction and release all
associated locks. The bug reported in DBAAS-7828 shows that when
autocommit is off, the locks are not released, resulting in
indefinite hangs on future attempts to change gtid_slave_pos. In
particular, the transaction was not properly finalized because
thd->server_status was not updated to reflect the end of the
transaction.

Solution:
========
This patch updates the code to properly commit the transaction after
reading gtid_slave_pos during mysqld start-up.

Reviewed By:
============
Andrei Elkin <andrei.elkin@mariadb.com>
2022-03-24 12:00:40 -06:00
Rucha Deodhar
0da5f45a50 MDEV-27819: func_2.xxx_charset skipped after renaming utf8 to utf8mb3
Analysis: utf8 character set is now utf8mb3 or utf8mb4. So charset_master
is not able to find utf8 at the beginning of test. Hence it skips the tests
that use charset_master.
Fix: rename utf8 to utf8mb3 in charset_master.
2022-03-24 20:30:04 +05:30
Marko Mäkelä
8684af76e3 MDEV-28137 Some memory transactions are unnecessarily complex
buf_page_get_zip(): Do not perform a system call inside a
memory transaction. Instead, if the page latch is unavailable,
abort the memory transaction and let the fall-back code path
wait for the page latch.

buf_pool_t::watch_remove(): Return the previous state of the block.

buf_page_init_for_read(): Use regular stores for moving the
buffer fix count of watch_remove() to the new block descriptor.

A more extensive version of this was reviewed by Daniel Black
and tested with Intel TSX-NI by Axel Schwenke and Matthias Leich.
My assumption that regular loads and stores would execute faster
in a memory transaction than operations like std::atomic::fetch_add()
turned out to be incorrect.
2022-03-24 16:09:04 +02:00
Brandon Nesterenko
174f1734a9 MDEV-14608: mysqlbinlog lastest backupfile size is 0
Problem:
========
When using mariadb-binlog with --raw and --stop-never, events from
the master's currently active log file should be written to their
respective log file specified by --result-file, and shown on-disk.
There is a bug where mariadb-binlog does not flush the result file
to disk when new events are received

Solution:
========
Add a function call to flush mariadb-binlog’s result file after
receiving an event in --raw mode.

Reviewed By:
============
Andrei Elkin <andrei.elkin@mariadb.com>
2022-03-24 07:40:29 -06:00
Brandon Nesterenko
32ab6219be MDEV-25580: rpl.rpl_semi_sync_slave_compressed_protocol crashes because of wrong packet
Problem:
========
When both semi-sync and slave compression are enabled, the numbering
on packet headers can become out of sync between the primary and
replica servers. More specifically, after the master flushes its
write, it should increment the counters that track packets. The
bug is such that the master only updates the normal packet counter
and leaves the compressed packet counter alone.

Solution:
========
After the master flushes, additionally increment the compressed
packet counter.

Reviewed By:
============
Andrei Elkin: <andrei.elkin@mariadb.com>
2022-03-24 07:25:22 -06:00
Marko Mäkelä
b101f19d29 MDEV-23974 fixup: rpl.rpl_gtid_stop_start fails
The call mtr.add_suppression() that was added
in commit 75b7cd680b
for MemorySanitizer and Valgrind runs is causing
a result difference for the test rpl.rpl_gtid_stop_start.
Let us disable the binlog for executing that statement.

Also, the test perfschema.statement_program_lost_inst
would fail due to the changes to have_innodb.inc in this commit.
To compensate for that, we will make more --suite=perfschema
tests run without InnoDB, and explicitly enable InnoDB in
those tests that depend on a transactional storage engine.
2022-03-24 13:43:58 +02:00
Daniel Black
0444d86baf Merge 10.7 2022-03-24 22:01:27 +11:00
Daniel Black
e86986a157 Merge 10.6 into 10.7 2022-03-24 18:57:07 +11:00
Marko Mäkelä
cf483a7766 MDEV-17441 fixup: Remove unused my_atomic long macros 2022-03-24 09:53:52 +02:00
Igor Babaev
bbf02c85ba MDEV-24281 Reading from freed memory when running main.view with --ps-protocol
This bug could affect prepared statements for the command CREATE VIEW with
specification that contained unnamed basic constant in select list. If
generation of a valid name for the corresponding view column required
resolution of conflicts with names of other columns that were explicitly
defined then execution of such prepared statement and following deallocation
of this statement led to reading from freed memory.

Approved by Oleksandr Byelkin <sanja@mariadb.com>
2022-03-23 12:50:50 -07:00
Sergei Golubchik
d2c019b200 MDEV-28107 S3 doesn't build if CURL is in non-default location 2022-03-23 17:42:15 +01:00
Sergei Golubchik
2ca3861b55 MDEV-28106 S3 tries to include thread.h while compiling on Windows
S3 engine and libmarias3 don't compile on Windows

also fixes:
MDEV-28104 Typo in storage/maria/s3_func.c: 'FN_DECVHAR
2022-03-23 17:42:15 +01:00
Marko Mäkelä
75b7cd680b MDEV-23974 Tests fail due to [Warning] InnoDB: Trying to delete tablespace
A few regression tests invoke heavy flushing of the buffer pool
and may trigger warnings that tablespaces could not be deleted
because of pending writes. Those warnings are to be expected
during the execution of such tests.

The warnings are also frequently seen with Valgrind or MemorySanitizer.
For those, the global suppression in have_innodb.inc does the trick.
2022-03-23 16:42:43 +02:00
Marko Mäkelä
44231dc6d5 Cleanup: have_sanitizer='ASAN,UBSAN'
This was suggested by Sergei Golubhick.
Fixes up commit b91a123d8c
2022-03-23 16:41:58 +02:00
Thirunarayanan Balathandayuthapani
c3ca729ba8 MDEV-27858 Assertion `page_dir_get_n_heap(new_page) == 2U' failed in PageBulk::init
- InnoDB should check whether bulk transaction id set to its own
transaction id before start bulk insert operation.

- When bulk insert failure happens, InnoDB should set the error info
of the transaction.
2022-03-23 10:15:52 +05:30
Tuukka Pasanen
9c57bbdad0 MDEV-28120 Remove false-positive Lintian linking error
Lintian uses objdump-command to determine is share object linked against
C-libary. Compression provider shared objects does not have any symbols
from C-library they only depend compression library and MariaDB
libaries. This is false-positive as Lintian error itself is correct as
shared object does not share any symbols from for example Glibc.

Added overrides suppress Lintian errors:
 * E: mariadb-plugin-provider-bzip2: library-not-linked-against-libc usr/lib/mysql/plugin/provider_bzip2.so
 * E: mariadb-plugin-provider-lz4: library-not-linked-against-libc usr/lib/mysql/plugin/provider_lz4.so
 * E: mariadb-plugin-provider-lzma: library-not-linked-against-libc usr/lib/mysql/plugin/provider_lzma.so
 * E: mariadb-plugin-provider-lzo: library-not-linked-against-libc usr/lib/mysql/plugin/provider_lzo.so
 * E: mariadb-plugin-provider-snappy: library-not-linked-against-libc usr/lib/mysql/plugin/provider_snappy.so
2022-03-23 13:49:24 +11:00
Tuukka Pasanen
770f62bf3e MDEV-28135: Add missing conflict and breaks for MariaDB 10.6
There is missing conflict and breaks for MariaDB 10.6 which
can cause upgrading from 10.6 not be clean and end up with
errors on Debian.
2022-03-23 13:34:07 +11:00
Tuukka Pasanen
1960f7b224 MDEV-27033: Clean lintian 'extended-description-is-empty' errors
Fix Debian lintian errors:
  E: mariadb-plugin-provider-bzip2: extended-description-is-empty
  E: mariadb-plugin-provider-lz4: extended-description-is-empty
  E: mariadb-plugin-provider-lzma: extended-description-is-empty
  E: mariadb-plugin-provider-lzo: extended-description-is-empty
  E: mariadb-plugin-provider-snappy: extended-description-is-empty
by adding Debian Policy Manual section 3.4 (The description
of a package) extended descriptions to mariadb-plugin-provide-*
packages
2022-03-23 13:14:00 +11:00
Ian Gilfillan
8153c974e6 Update contributors 2022-03-23 10:47:27 +11:00
Andrei
5ccd845d51 MDEV-27760 event may non stop replicate in circular semisync setup
MDEV-21117 had to relax own events acceptance condition for a case
when a former semisync master server recovers after crash as the
semisync slave. That however admitted a possibility for endless event
"orbiting" in the non-strict slave gtid mode of semisync circular
setup.

The same server-id event termination is restored now for
the non-strict gtid mode to follow regular rules (that is it's ignored
unless @@global.replicate_same_server_id allows it in).

To address MDEV-21117 recovery agenda,
in the strict gtid mode and the transaction's gtid ordered strictly
greater than the current slave gtid state, the same server-id
transaction is accepted.

The gtid strict mode is safe to accept transactions even if
the slave state were not set correct by the user, e.g
at the former master.
An added test shows a typical out-of-order error at execution so
no data corruption is guaranteed in such a case.
2022-03-22 19:20:19 +02:00
Alexander Barkov
0c4c064f98 MDEV-27743 Remove Lex::charset
This patch also fixes:

MDEV-27690 Crash on `CHARACTER SET csname COLLATE DEFAULT` in column definition
MDEV-27853 Wrong data type on column `COLLATE DEFAULT` and table `COLLATE some_non_default_collation`
MDEV-28067 Multiple conflicting column COLLATE clauses are not rejected
MDEV-28118 Wrong collation of `CAST(.. AS CHAR COLLATE DEFAULT)`
MDEV-28119 Wrong column collation on MODIFY + CONVERT
2022-03-22 17:12:15 +04:00
Alexander Barkov
d25b10fede MDEV-27712 Reduce the size of Lex_length_and_dec_st from 16 to 8
User visible change:
Removing the length specified by user from error messages:
ER_TOO_BIG_SCALE and ER_TOO_BIG_PRECISION
as discussed with Sergei.
2022-03-22 14:42:54 +04:00
Jan Lindström
cade21b4e0 MDEV-27775 : Some Galera tests fail on FreeBSD due to "unknown signal 9"
Replace 9 with KILL
2022-03-22 09:27:43 +02:00
Jan Lindström
8f4d7e365e MDEV-27775 : Some Galera tests fail on FreeBSD due to "unknown signal 9"
Replace 9 with KILL
2022-03-22 08:25:04 +02:00
Julius Goryavsky
ab1a792571 MDEV-26971 post-fix: fixes for SST scripts
1) Fixed a bug with incorrect calculation of the payload length
   when using the --innodb-undo-directory option on the command
   line (not in the .cnf file);
2) Fixed a bug with incorrect processing of progress=none in the
   presence of *.qp-files;
3) Progress processing for *.qp-files has been made more logically
   unified with the semantics of the adjust_progress function;
4) Added forgotten option "--maxdepth 1" when searching for
   *.qp-files;
5) Eliminated "cosmetic" differences with the SST scripts in
   previous versions, which otherwise will cause problems when
   automatically migrating future changes;
6) Fixed a bug caused by automatic migration of the MDEV-27524.
2022-03-22 04:51:39 +01:00
Julius Goryavsky
35725df6e2 MDEV-27524 addendum: fix for bug introduced by automatic migration 2022-03-22 03:23:32 +01:00
Alexander Barkov
0812d0de8d MDEV-28131 Unexpected warning while selecting from information_schema.processlist
Problem:

DECIMAL columns in I_S must be explicitly set of some value.

I_S columns do not have `DEFAULT 0` (after MDEV-18918), so during
restore_record() their record fragments pointed by Field::ptr are
initialized to zero bytes 0x00.
But an array of 0x00's is not a valid binary DECIMAL value.
So val_decimal() called for such Field_new_decimal generated a warning
when seeing a wrong binary encoded DECIMAL value in the record.

Fix:

Explicitly setting INFORMATION_SCHEMA.PROCESSLIST.PROGRESS
to the decimal value of 0 if no progress information is available.
2022-03-21 16:42:58 +04:00
Alexey Botchkov
6277e7df6b MDEV-22742 UBSAN: Many overflow issues in strings/decimal.c - runtime error: signed integer overflow: x * y cannot be represented in type 'long long int' (on optimized builds).
Avoid integer overflow, do the check before the calculation.
2022-03-21 15:05:42 +04:00
Sergei Golubchik
f54d6380d2 MDEV-27980 file-key-management plugin disabled in mysql_install_db breaks automated deployments (and container initialization)
fix a 2015 typo in build scripts.

--without-plugin=plugin_file_key_management translates to
-DPLUGIN_PLUGIN_FILE_KEY_MANAGEMENT=NO

replace it with a line from 10.4 that builds the plugin
dynamically.
2022-03-21 11:01:40 +01:00
Oleksandr Byelkin
fbc1cc974e MDEV-26009 Server crash when calling twice procedure using FOR-loop
The problem was that instructions sp_instr_cursor_copy_struct and
sp_instr_copen uses the same lex, adding and removing "tail" of
prelocked tables and forgetting that tail of all tables is kept in
LEX::query_tables_last. If the LEX used only by one instruction
or the query do not have prelocked tables it is not important.
But to work correctly in all cases LEX::query_tables_last should
be reset to make new tables added in the correct list (after last
table in the LEX instead after last table of the prelocking "tail"
which was cut).
2022-03-21 07:55:57 +01:00
Sergei Golubchik
421153848f MDEV-27980 file-key-management plugin disabled in mysql_install_db breaks automated deployments (and container initialization)
Revert "Silence the file-key-management plugin during mysql_install_db"

This reverts commit e99d3da638.
2022-03-20 21:38:48 +01:00
Sutou Kouhei
cf86580f2b MDEV-28032 "git submodule update --depth 1" may fail with old Git
submodules.cmake: don't use "--depth 1" with old Git

Old Git may not work with "--depth 1" when the referenced commit hash
is far from HEAD.

Newer Git improves the situation. For example:
fb43e31f2b

It's safe to not use "--depth 1" with old Git.

Closes 
2022-03-19 09:38:57 +01:00
Jan Lindström
12ce9b4f02 Fix compile error. 2022-03-18 20:50:10 +01:00
Alexey Yurchenko
a686c10e87 MDEV-26971: Implement progress reporting by mariabackup SST script
Add option to explicitly disable progress reporting.

Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>
2022-03-18 20:50:10 +01:00
Alexey Yurchenko
eceb9e2478 MDEV-26971: JSON file interface to wsrep node state.
Fix status reporting - move it from dead code to actually executing.

Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>
2022-03-18 20:50:10 +01:00
Jan Lindström
7878eae95e Small fixes
Add new requirement for pv tool for debian and rpm
  Fix one test result difference.
  Set message about missing progress reporting tool pv as info.
2022-03-18 20:50:10 +01:00
Alexey Yurchenko
73d80c8672 MDEV-26971: Implement progress reporting by mariabackup SST script
Currently covers network transfer stage on donor and joiner.

Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>
2022-03-18 16:38:41 +01:00
Alexey Yurchenko
98355a0789 MDEV-26971: Support for progress reporting from SST scripts.
New feedback events:
 - "total N": signals new SST stage and reports estimated total work
 - "complete N": reports completed work in the current stage

Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>
2022-03-18 16:38:41 +01:00
Alexey Yurchenko
9d7e596ba6 MDEV-26971: JSON file interface to wsrep node state.
Integration with status reporter in wsrep-lib.

Status reporter reports changes in wsrep state and logged errors/
warnings to a json file which then can be read and interpreted by
an external monitoring tool.

Rationale: until the server is fully initialized it is unaccessible
by client and the only source of information is an error log which
is not machine-friendly. Since wsrep node can spend a very long time
in initialization phase (state transfer), it may be a very long time
that automatic tools can't easily monitor its liveness and progression.

New variable: wsrep_status_file specifies the output file name.
If not set, no file is created and no reporting is done.

Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>
2022-03-18 16:38:41 +01:00
Jan Lindström
d526551587 Update wsrep-lib submodule 2022-03-18 16:38:41 +01:00
Marko Mäkelä
f6fcf827b3 MDEV-28111 fixup: Correct the start-up message
By default, we should say "InnoDB: Buffered log writes" instead of
"InnoDB: File system buffers for log disabled" if we are able to
determine the physical block size on Linux.
2022-03-18 15:27:57 +02:00
Monty
74e668eaeb Fixed warning for maria.maria-recovery2 about crashed table
The bug was a missing va_start in eprint() which caused a wrong table
name to be printed.
Patch backported from 10.3.
2022-03-18 13:26:50 +02:00
Sergei Golubchik
10d9b890b0 Merge branch '10.8' into 10.9 2022-03-18 11:14:48 +01:00
Marko Mäkelä
8840583a92 MDEV-27909 InnoDB: Failing assertion: state == TRX_STATE_NOT_STARTED ... on DDL
The fix in commit 6e390a62ba (MDEV-26772)
was a step to the right direction, but implemented incorrectly.
When an InnoDB persistent statistics table cannot be locked immediately,
we must not let row_mysql_handle_errors() to roll back the transaction.

lock_table_for_trx(): Add the parameter no_wait (default false)
for an immediate return of DB_LOCK_WAIT in case of a conflict.

ha_innobase::delete_table(), ha_innobase::rename_table():
Pass no_wait=true to lock_table_for_trx() when needed,
instead of temporarily setting THDVAR(thd, lock_wait_timeout) to 0.
2022-03-18 10:52:08 +02:00
Sergei Golubchik
5c11e7eead update test results 2022-03-18 09:47:58 +01:00
Jan Lindström
c519aa3d7a MDEV-24143 : Galera nodes "randomly" crashing in Item_func_release_lock::val_int
Fixed on MDEV-27713. Added additional test case.
2022-03-18 08:30:26 +02:00