Commit graph

200263 commits

Author SHA1 Message Date
Brandon Nesterenko
c37b2087b4 MDEV-7850: Extend GTID Binlog Events with Thread Id
This patch augments Gtid_log_event with the user thread-id.
In particular that compensates for the loss of this info in
Rows_log_events.

Gtid_log_event::thread_id gets visible in mysqlbinlog output like

   16:21:45 server id 1  end_log_pos 537 CRC32 0x1cf1d963  GTID 0-1-2 ddl thread_id=10

as 64 bit unsigned integer.

While the size of Gtid event has grown by 8-9 bytes
replication from OLD <-> NEW is not affected by it.

This work was started by the late Sujatha Sivakumar.
Brandon Nesterenko took it over, reviewed initial patches and extended
the work.

Reviewed-by: <andrei.elkin@mariadb.com>
2024-01-22 07:24:22 -07:00
Brandon Nesterenko
207c85783b MDEV-33283: Binlog Checksum is Zeroed by Zlib if Part of Event Data is Empty
An existing binlog checksum can be overridden to 0 if writing a NULL
payload when using Zlib for the computation. That is, calling into
Zlib's crc32 with empty data initializes an incremental CRC
computation to 0.

This patch changes the Log_event_writer::write_data() to exit
immediately if there is nothing to write, thereby bypassing the
checksum computation. This follows the pattern of
Log_event_writer::encrypt_and_write(), which also exits immediately
if there is no data to write.

Reviewed By:
============
Andrei Elkin <andrei.elkin@mariadb.com>
2024-01-22 07:15:42 -07:00
Vicențiu Ciorbaru
3cd8875145
MDEV-14448: Ctrl-C should not exit the client
This patch introduces the following behaviour for Linux while
maintaining old behaviour for Windows:

Ctrl + C (sigint) clears the current buffer and redraws the prompt.
Ctrl-C no longer exits the client if no query is running.
Ctrl-C kills the current running query if there is one. If there is an
error communicating with the server while trying to issue a KILL QUERY,
the client exits. This is in line with the past behaviour of Ctrl-C.
On Linux Ctrl-D can be used to close the client.
On Windows Ctrl-C and Ctrl-BREAK still exits the client if no query is running.
Windows can also exit the client via \q<enter> or exit<enter>.

== Implementation details ==
The Linux implementation has two corner cases, based on which library is
used: libreadline or libedit, both are handled in code to achieve the
same user experience.

Additional code is taken from MySQL, ensuring there is identical
behaviour on Windows, to MySQL's mysql client implementation for other
CTRL- related signals.

* The CTRL_CLOSE, CTRL_LOGOFF, CTRL_SHUTDOWN will issue the equivalent
  of CTRL-C and "end" the program. This ensures that the query is killed
  when the client is closed by closing the terminal, logging off the
  user or shutting down the system. The latter two signals are not sent
  for interactive applications, but it handles the case when a user has
  defined a service to use mysql client to issue a command. See
  https://learn.microsoft.com/en-us/windows/console/handlerroutine

This patch is built on top of the initial work done by Anel Husakovic
<anel@mariadb.org>.
Closes 
2024-01-22 14:59:48 +02:00
Marko Mäkelä
1acf6a0f84 MDEV-14425 fixup: mariabackup.huge_lsn,strict_crc32 rdiff
Some old versions of "patch" (such as patch 2.5.9 on Microsoft Windows)
require that a file name header be present.
To ensure that the diff will be applied, let us add the header.
2024-01-22 12:42:37 +02:00
Marko Mäkelä
0335629eb3 MDEV-32242 fixup: innodb.doublewrite test may be skipped again
It is not sufficient to sleep for only 1 second to
ensure that the page cleaner has gone idle.
The timings could have been changed compared to earlier releases
by commit a635c40648 (MDEV-27774).
Let us allow the non-debug test innodb.doublewrite to be skipped,
and remove the insufficient 1-second sleep.
2024-01-22 11:31:22 +02:00
Marko Mäkelä
35cc4b6c05 Merge 10.11 into 11.0 2024-01-22 10:10:50 +02:00
Marko Mäkelä
7f11fad85a MDEV-32968: After-merge fix
This fixes up merge commit 9d20853c74
2024-01-22 10:04:11 +02:00
Marko Mäkelä
b3ca7fa089 Merge 10.6 into 10.11 2024-01-22 08:49:04 +02:00
Marko Mäkelä
495e7f1b3d MDEV-33053 fixup: Correct a condition before a message 2024-01-22 08:24:08 +02:00
Marko Mäkelä
5c243d4caf Merge 10.5 into 10.6 2024-01-22 08:20:08 +02:00
Marko Mäkelä
e237925963 MDEV-33031 test fixup for HAVE_PERFSCHEMA=NO 2024-01-22 08:19:00 +02:00
Daniel Black
0c23f84d8d MDEV-32983 cosmetic improvement on path separator near ib_buffer_pool
A mix of path separators looks odd.

  InnoDB: Loading buffer pool(s) from C:\xampp\mysql\data/ib_buffer_pool

This was changed in cf552f5886

Both forward slashes and backward slashes work on Windows. We do not
use \\?\ names.

So we improve the consistent look of it so it doesn't look like a bug.

Normalize, in this case, the path separator to \ for making the filename.

Reported thanks to Github user @celestinoxp.

Closes: https://github.com/ApacheFriends/xampp-build/issues/33
Reviewed by: Marko Mäkelä and Vladislav Vaintroub
2024-01-22 16:56:00 +11:00
Sisi Huang
7e8e51eb3a MDEV-32990 federatedx time_zone round trips
Modified `federatedx_io_mysql::actual_query` to set the time zone to '+00:00' only upon establishing a new connection instead of with each query execution.
2024-01-22 13:44:53 +11:00
Sergei Golubchik
3a33ae8601 MDEV-33091 pcre2 headers aren't found on Solaris
use pkg-config to find pcre2, if possible

rename PCRE_INCLUDES to use PKG_CHECK_MODULES naming, PCRE_INCLUDE_DIRS
2024-01-19 20:23:45 +01:00
Igor Babaev
e8041c7065 MDEV-33270 Failure to call SP invoking another SP with parameter requiring type conversion
This patch corrects the fix for MDEV-32569. The latter has not taken into
account the fact not each statement uses the SELECT_LEX structure. In
particular CALL statements do not use such structure. However the parameter
passed to the stored procedure used in such a statement may require an
invocation of Type_std_attributes::agg_item_set_converter().

Approved by Oleksandr Byelkin <sanja@mariadb.com>
2024-01-19 08:38:00 -08:00
Thirunarayanan Balathandayuthapani
7573fe8b07 MDEV-32968 InnoDB fails to restore tablespace first page from doublewrite buffer when page is empty
recv_dblwr_t::find_first_page(): Free the allocated memory
to read the first 3 pages from tablespace.

innodb.doublewrite: Added sleep to ensure page cleaner thread
wake up from my_cond_wait
2024-01-19 17:01:36 +05:30
Alexander Barkov
985e3dfc05 MDEV-33182 Server assertion fails when trying to test the connection with DBeaver
Some connectors (JDBC, Node.js) can sent non-default collation IDs
in the handshake packet. The code in thd_init_client_charset() handling
@@character_set_collations did not expect that and crashed on DBUG_ASSERT.

Changing the code to ignore @@character_set_collations in case of non-default
IDs. This fixes the problem in a backward compatible
(with pre-@@character_set_collations server versions) way for such connectors
sending non-default IDs.
2024-01-19 14:48:16 +04:00
Marko Mäkelä
21560bee9d Revert "MDEV-32899 InnoDB is holding shared dict_sys.latch while waiting for FOREIGN KEY child table lock on DDL"
This reverts commit 569da6a7ba,
commit 768a736174, and
commit ba6bf7ad9e
because of a regression that was filed as MDEV-33104.
2024-01-19 12:46:11 +02:00
Marko Mäkelä
7e65e3027e MDEV-33275 buf_flush_LRU(): mysql_mutex_assert_owner(&buf_pool.mutex) failed
In commit a55b951e60 (MDEV-26827)
an error was introduced in a rarely executed code path of the
buf_flush_page_cleaner() thread. As a result, the function
buf_flush_LRU() could be invoked while not holding buf_pool.mutex.

Reviewed by: Debarun Banerjee
2024-01-19 12:40:32 +02:00
Marko Mäkelä
d34479dc66 MDEV-33053 InnoDB LRU flushing does not run before running out of buffer pool
buf_flush_LRU(): Display a warning if no pages could be evicted and
no writes initiated.

buf_pool_t::need_LRU_eviction(): Renamed from buf_pool_t::ran_out().
Check if the amount of free pages is smaller than innodb_lru_scan_depth
instead of checking if it is 0.

buf_flush_page_cleaner(): For the final LRU flush after a checkpoint
flush, use a "budget" of innodb_io_capacity_max, like we do in the
case when we are not in "furious" checkpoint flushing.

Co-developed by: Debarun Banerjee
Reviewed by: Debarun Banerjee
Tested by: Matthias Leich
2024-01-19 12:40:16 +02:00
Oleksandr Byelkin
16f2f8e5a7 new CC 3.3 2024-01-19 08:44:32 +01:00
Marko Mäkelä
9374772ecd Merge 10.11 into 11.0 2024-01-19 09:07:48 +02:00
Daniel Black
82e8633420 innodb: IO Error message missing space
Noted by Susmeet Khaire - thanks.
2024-01-19 17:33:44 +11:00
Brad Smith
2ef01d0034 wsrep scripts fixes for working on OpenBSD 2024-01-19 15:45:08 +11:00
Marko Mäkelä
a6290a5bc5 MDEV-33095 innodb_flush_method=O_DIRECT creates excessive errors on Solaris
The directio(3C) function on Solaris is supported on NFS and UFS
while the majority of users should be on ZFS, which is a copy-on-write
file system that implements transparent compression and therefore
cannot support unbuffered I/O.

Let us remove the call to directio() and simply treat
innodb_flush_method=O_DIRECT in the same way as the previous
default value innodb_flush_method=fsync on Solaris. Also, let us
remove some dead code around calls to os_file_set_nocache() on
platforms where fcntl(2) is not usable with O_DIRECT.

On IBM AIX, O_DIRECT is not documented for fcntl(2), only for open(2).
2024-01-19 15:34:33 +11:00
Sophist
ac774a2656 Replace incorrect message mariadb-safe with correct mariadbd-safe 2024-01-19 14:56:25 +11:00
Sophist
a2eb664f08 Replace incorrect message mariadb-safe with correct mariadbd-safe 2024-01-19 14:56:25 +11:00
Marko Mäkelä
9d20853c74 Merge 10.6 into 10.11 2024-01-18 19:22:23 +02:00
Libing Song
8bf9f21855 MDEV-32894 mysqlbinlog flashback support binlog_row_image FULL_NODUP mode
Summary
=======
With FULL_NODUP mode, before image inclues all columns and after
image inclues only the changed columns. flashback will swap the
value of changed columns from after image to before image.
For example:
  BI: c1, c2, c3_old, c4_old
  AI: c3_new, c4_new
flashback will reconstruct the before and after images to
  BI: c1, c2, c3_new, c4_new
  AI: c3_old, c4_old

Implementation
==============
When parsing the before and after image, position and length of
the fields are collected into ai_fields and bi_fields, if it is an
Update_rows_event and the after image doesn't includes all columns.

The changed fields are swapped between bi_fields and ai_fields.
Then it recreates the before image and after image by using
bi_fields and ai_fields. nullbit will be set to 1 if the
field is NULL, otherwise nullbit will be 0.

It also optimized flashback a little bit.
- calc_row_event_length is used instead of print_verbose_one_row
- swap_buff1 and swap_buff2 are removed.
2024-01-18 12:26:21 +00:00
Marko Mäkelä
ee1407f74d MDEV-32268: GNU libc posix_fallocate() may be extremely slow
os_file_set_size(): Let us invoke the Linux system call fallocate(2)
directly, because the GNU libc posix_fallocate() implements a fallback
that writes to the file 1 byte every 4096 or fewer bytes. In one
environment, invoking fallocate() directly would lead to 4 times the
file growth rate during ALTER TABLE. Presumably, what happened was
that the NFS server used a smaller allocation block size than 4096 bytes
and therefore created a heavily fragmented sparse file when
posix_fallocate() was used. For example, extending a file by 4 MiB
would create 1,024 file fragments. When the file is actually being
written to with data, it would be "unsparsed".

The built-in EOPNOTSUPP fallback in os_file_set_size() writes a buffer
of 1 MiB of NUL bytes. This was always used on musl libc and other
Linux implementations of posix_fallocate().
2024-01-18 11:00:27 +02:00
Marko Mäkelä
f63045b119 MDEV-33213 fixup: GCC 5 -Wconversion 2024-01-18 10:14:21 +02:00
Rucha Deodhar
468d29f893 remove the accidentally pushed temp.test file 2024-01-18 11:58:50 +05:30
Sophist
c95ba183d2 Replace incorrect message mariadb-safe with correct mariadbd-safe 2024-01-18 08:51:17 +11:00
Vladislav Vaintroub
03854a84ab MDEV-32374 Improve lsn_lock. Also use futex-like on Windows
Upon further benchmarking, it turns out srw_mutex performs overall
slightly better with WaitOnAddress than CRITICAL_SECTION.
2024-01-17 22:43:42 +01:00
Marko Mäkelä
ad13fb36bf Merge 10.6 into 10.11 2024-01-17 17:37:15 +02:00
Andrew Hutchings
f552febe43 MDEV-30879 Add support for up to BASE 62 to CONV()
BASE 62 uses 0-9, A-Z and then a-z to give the numbers 0-61. This patch
increases the range of the string functions to cover this.

Based on ideas and tests in PR , but re-written into the charset
functions.

Includes fix by Sergei, UBSAN complained:
ctype-simple.c:683:38: runtime error: negation of -9223372036854775808
cannot be represented in type 'long long int'; cast to an unsigned
type to negate this value to itself

Co-authored-by: Weijun Huang <huangweijun1001@gmail.com>
Co-authored-by: Sergei Golubchik <serg@mariadb.org>
2024-01-17 15:24:26 +00:00
Robin Newhouse
615f4a8c9e MDEV-32587 Allow json exponential notation starting with zero
Modify the NS_ZERO state in the JSON number parser to allow
exponential notation with a zero coefficient (e.g. 0E-4).

The NS_ZERO state transition on 'E' was updated to move to the
NS_EX state rather than returning a syntax error. Similar change
was made for the NS_ZE1 (negative zero) starter state.

This allows accepted number grammar to include cases like:

- 0E4
- -0E-10

which were previously disallowed. Numeric parsing remains
the same for all other states.

Test cases are added to func_json.test to validate parsing for
various exponential numbers starting with zero coefficients.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services.
2024-01-17 19:25:43 +05:30
Marko Mäkelä
3a96eba25f Merge 10.5 into 10.6 2024-01-17 13:35:05 +02:00
Marko Mäkelä
6a514ef672 MDEV-30940: Try to fix the test 2024-01-17 12:50:44 +02:00
Oleksandr Byelkin
8e337e016f new WolfSSL v5.6.6-stable 2024-01-17 10:45:05 +01:00
Marko Mäkelä
f8c88d905b MDEV-33213 History list is not shrunk unless there is a pause in the workload
The parameter innodb_undo_log_truncate=ON enables a multi-phased logic:
1. Any "producers" (new starting transactions) are prohibited
from using the rollback segments that reside in the undo tablespace.
2. Any transactions that use any of the rollback segments must be
committed or aborted.
3. The purge of committed transaction history must process all the
rollback segments.
4. The undo tablespace is truncated and rebuilt.
5. The rollback segments are re-enabled for new transactions.

There was one flaw in this logic: The first step was not being invoked
as often as it could be, and therefore innodb_undo_log_truncate=ON
would have no chance to work during a heavy write workload.

Independent of innodb_undo_log_truncate, even after
commit 86767bcc0f
we are missing some chances to free processed undo log pages.
If we prohibited the creation of new transactions in one busy
rollback segment at a time, we would be eventually guaranteed
to be able to free such pages.

purge_sys_t::skipped_rseg: The current candidate rollback segment
for shrinking the history independent of innodb_undo_log_truncate.

purge_sys_t::iterator::free_history_rseg(): Renamed from
trx_purge_truncate_rseg_history(). Implement the logic
around purge_sys.m_skipped_rseg.

purge_sys_t::truncate_undo_space: Renamed from truncate.

purge_sys.truncate_undo_space.last: Changed the type to integer
to get rid of some pointer dereferencing and conditional branches.

purge_sys_t::truncating_tablespace(), purge_sys_t::undo_truncate_try():
Refactored from trx_purge_truncate_history().
Set purge_sys.truncate_undo_space.current if applicable,
or return an already set purge_sys.truncate_undo_space.current.

purge_coordinator_state::do_purge(): Invoke
purge_sys_t::truncating_tablespace() as part of the normal work loop,
to implement innodb_undo_log_truncate=ON as often as possible.

trx_purge_truncate_rseg_history(): Remove a redundant parameter.

trx_undo_truncate_start(): Replace dead code with a debug assertion.

Correctness tested by: Matthias Leich
Performance tested by: Axel Schwenke
Reviewed by: Debarun Banerjee
2024-01-17 11:14:24 +02:00
Ian Gilfillan
b697e25adc Update 11.2 HELP 2024-01-17 18:33:27 +11:00
Ian Gilfillan
83a79ba33b Update 11.3 HELP 2024-01-17 18:07:26 +11:00
Yuchen Pei
9c059a4f1c
Spider: no need to check for ubsan when running ubsan tests
It's ok to run these tests without ubsan too, and we get some tests
for free.
2024-01-17 10:33:02 +11:00
Alexander Barkov
fa3171df08 MDEV-27666 User variable not parsed as geometry variable in geometry function
Adding GEOMETRY type user variables.
2024-01-16 18:53:23 +04:00
Yuchen Pei
931df937e9
MDEV-32559 failing spider signal_ddl_recovery_done callback should result in spider deinit
Since 0930eb86cb, system table creation
needed for spider init is delayed to the signal_ddl_recovery_done
callback. Since it is part of the init, failure should result in
spider deinit.

We also remove the call to spider_init_system_tables() from
spider_db_init(), as it was removed in the commit mentioned above and
accidentally restored in a merge.
2024-01-16 17:17:50 +11:00
Ian Gilfillan
3b32110ac4 Update 11.1 HELP 2024-01-16 15:33:30 +11:00
Thirunarayanan Balathandayuthapani
caad34df54 MDEV-32968 InnoDB fails to restore tablespace first page from doublewrite buffer when page is empty
- InnoDB fails to find the space id from the page0 of
the tablespace. In that case, InnoDB can use
doublewrite buffer to recover the page0 and write
into the file.

- buf_dblwr_t::init_or_load_pages(): Loads only the pages
which are valid.(page lsn >= checkpoint). To do that,
InnoDB has to open the redo log before system
tablespace, read the latest checkpoint information.

recv_dblwr_t::find_first_page():
1) Iterate the doublewrite buffer pages and find the 0th page
2) Read the tablespace flags, space id from the 0th page.
3) Read the 1st, 2nd and 3rd page from tablespace file and
compare the space id with the space id which is stored
in doublewrite buffer.
4) If it matches then we can write into the file.
5) Return space which matches the pages from the file.

SysTablespace::read_lsn_and_check_flags(): Remove the
retry logic for validating the first page. After
restoring the first page from doublewrite buffer,
assign tablespace flags by reading the first page.

recv_recovery_read_max_checkpoint(): Reads the maximum
checkpoint information from log file

recv_recovery_from_checkpoint_start(): Avoid reading
the checkpoint header information from log file

Datafile::validate_first_page(): Throw error in case
of first page validation fails.
2024-01-15 14:08:27 +05:30
Thirunarayanan Balathandayuthapani
653cb195d3 MDEV-26740 Inplace alter rebuild increases file size
PageBulk::init(): Unnecessary reserves the extent before
allocating a page for bulk insert. btr_page_alloc()
capable of handing the extending of tablespace.
2024-01-15 13:04:10 +05:30
Tuukka Pasanen
ee30491e50 MDEV-32111: Debian Sid/Trixie will not have libncurses 5 anymore
Upstream Debian Sid which will become Debian Trixie (13)
have dropped NCurses version 5 and changed dev package name just
libncurses-dev
2024-01-14 13:27:10 +11:00