relax the assert, allowing '\n' at the end if the string is exactly
MYSQL_ERRMSG_SIZE-1 bytes long. It likely doesn't end with '\n' but
was truncated at the middle.
also, use MYSQL_ERRMSG_SIZE in my_error.c not a separate define
that must be "kept in sync"
mariadb from a git tree
Problems:
1. bison incorrectly issues a warning when it sees "b4_bin"
2. MacOSX linker issues a warning about duplicate mysys/dbug/etc libraries.
On 1: worked around by adding a compiler define with a different name
aliasing the character set variable used with a name that
won't trigger the bison warning
On 2: This is due to the fact that there's a circular dependency between
mysys and dbug (among others). Turned the warning off by adding a compiler
option to be applied to targets lined with mysys to turn the warning off.
The binary log could be corrupted when committing a large transaction
(i.e. one whose data exceeds the binlog_cache_size limit and spills
into a tmp file) in binlog_format=row if the server's --tmp-dir is
full. The corruption that happens is only the GTID of the errored
transaction would be written into the binary log, without any
body/finalizing events. This would happen because the content of the
transaction wasn't flushed at the proper time, and the transaction's
binlog cache data was not durable while trying to copy the content
from the binlog cache file into the binary log itself. While switching
the tmp file from a WRITE_CACHE to a READ_CACHE, the server would see
there is still data to flush in the cache, and first try to flush it.
This is not a valid time to flush that data to the temporary file
though, as:
1. The GTID event has already been written directly to the binary
log. So if this flushing fails, it leaves the binary log in a
corrupted state.
2. This is done during group commit, and will slow down other
concurrent transactions, which are otherwise ready to commit.
This patch fixes these issues by ensuring all transaction data is
fully flushed to its temporary file (if used) before starting any
critical paths, i.e. in binlog_flush_cache(). Note that if the binlog
cache is solely in-memory, this flush-to-temporary-file is skipped.
Reviewed-by: Andrei Elkin <andrei.elkin@mariadb.com>
Signed-off-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
This is useful if one has MariaDB config files in /etc that should only be
visible for the super user and the MariaDB server but one still want to
provide options for normal users, like skip-ssl.
Fix my_print_help() to print "No currently supported values".
It used to print the first value even if it was hidden.
(cherry picked from commit 8664461e80)
@@new_mode is a set of flags to control introduced features.
Flags are by default off. Setting a flag in @@new_mode will introduce
a new different server behaviour and/or set of features.
We also introduce a new option 'hidden_values' into some system variable
types to hide options that we do not wish to show as options.
- Don't print hidden values in mysqld --help output.
- Make get_options() use the same logic as check_new_mode_value() does.
- Setting @@new_mode=ALL shouldn't give warnings.
(cherry picked from commit f7387cb13d)
When binlog_row_image=MINIMAL, UPDATE_ROWS_EVENT may change columns
that are not in the before image. Such columns had their bit set in
table->write_set, but was missing their bit in table->read_set.
As part of this patch, bitmap_union() is extended to handle bitmaps of
different sizes, similar to bitmap_intersect().
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
For the algorithms in crc32_arm64.c to work consistenly, all multibyte
integers loaded from the input buffer have to be loaded in little-endian.
Tested on an aarch64_be-unknown-linux-musl system, with:
ninja crc32-t && unittest/mysys/crc32-t
Test results:
unpatched:
+------------------------------------+-------------------+--------------------+
| Code path | GCC 15.2.0 | Clang 20.1.8 |
+------------------------------------+-------------------+--------------------+
|`crc32c_aarch64` | 25 failed tests | 25 failed tests |
+------------------------------------+-------------------+--------------------+
|`crc32c_aarch64_pmull` (assembly) | 25 failed tests | doesn't compile |
+------------------------------------+-------------------+--------------------+
|`crc32c_aarch64_pmull` (intrinsics) | 25 failed tests | 26 failed tests |
+------------------------------------+-------------------+--------------------+
patched:
+------------------------------------+-------------------+--------------------+
| Code path | GCC 15.2.0 | Clang 20.1.8 |
+------------------------------------+-------------------+--------------------+
|`crc32c_aarch64` | success | success |
+------------------------------------+-------------------+--------------------+
|`crc32c_aarch64_pmull` (assembly) | success | doesn't compile |
+------------------------------------+-------------------+--------------------+
|`crc32c_aarch64_pmull` (intrinsics) | success | 1 failed test |
+------------------------------------+-------------------+--------------------+
Implementation notes:
- uintNkorr uses the byte-wise implementation of a fixed-endian load,
which can't always be turned back into a single load instruction.
On aarch64-unknown-linux-musl (little endian) with GCC 15.2, this
resulted in a code pessimization at -O0 and -O1, but -Os and -O2 are
fine, so this is not a problem.
On MSVC/Windows there does appear to be a pessimization[1].
- Using le16toh etc. are a potential alternative, as these macros are
no-ops on little-endian platforms that support them. The endian(3)
manpage suggests that NetBSD, FreeBSD, and glibc (in addition to
musl), but not OpenBSD, support them[2]. Windows support is unclear.
leNNtoh has the added disadvantage of being unusual in the MariaDB
codebase, so it's better to stick with uintNkorr.
[1]: https://godbolt.org/z/oncv9Gj19
[2]: https://man.he.net/man3/htole64
This was generally good to get done but also needed to be able to run
mariabackup test under asan.
Things freed:
- Allocated variables (mysql_tmpdir_list, opt_passwd etc)
- InnoDB variables
- Results from SQL queries (A lot of sql queries did not free their result)
- Allocated sys_vars
- Server variables (mysql_server_end())
- Memory allocated by plugins (encryption)
- Free variables allocated by my_default. (Old code had a bug that caused
these to not be freed)
Other things:
- Moved freeing of mysql_tmpdir_list to main, as the old code did not
free the last mysqltmp_dir allocation. Now we also initialize the
variable only once.
- Fixed a serious, potentially 'crashing at end' bug where we called
free_defaults() with wrong pointers.
- Fixed a bug related to update_malloc_size() where we did not take
into account the it was not changed.
- Fixed a bug in Sys_var_charptr_base where we did not allocate
default values. This could lead to trying to free not allocated values
in xtrabackup.
- Added sf_have_memory_leak() to be able to easily check if there was
a memory leak when using safemalloc()
- sf_report_leaked_memory() now returns 1 if a memory leak was found.
It turns out that there is no need to use the evex512 target attribute
that had been introduced in GCC 14 and clang 18, and will be
removed in GCC 16.
At the time when the evex512 attribute was introduced, Intel had plans
to release products with AVX10.1-256, supporting the AVX512 instruction set
but limited to 256-bit registers. The evex512 attribute would have been
needed to "opt in" to the 512-bit instruction variants (using the EVEX
instruction prefix). Later on, Intel revised its plans to make AVX10.1
always 512-bit.
Reviewed by: Kristian Nielsen
The code was mistakenly using _mm512_castsi128_si512() to convert
__m128i to __m512i, which leaves the upper bits undefined. Use instead
_mm512_zextsi128_si512() which zero-extends as needed.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>
The problem was a bug in my_qsort from 1980 where we converted
difference of pointers to int instead of longlong.
This causes sort to fail if the size of the sort array > 2147483648 bytes.
In case of myisam recovery, we are sorting pointers, which limited the max
number of elements to 268435456.
In the case of a longlong and a 6 byte pointers, this equals to a
sort buffer of about 5G.
Fixed by changing the cast to longlong.
The AVX512 accelerated CRC-32 computation that had been added
in commit 9ec7819c58 was disabled
in commit a293dfd92a for GCC 16.
Let us enable that logic by applying
dr-m/crc32_simd@075bacb0cc
which makes use of the avx10.1 target attribute that had been
introduced in GCC 15.
Recently, evex512 was removed from gcc trunk [1] which will eventually
become gcc 16, and that leads to a build failure in
mariadb, originally reported downstream in a Gentoo bug [2].
This is reproducible across all versions from 10.6 to current master.
The change is as simple as adding an upper boundary to which
gcc versions can use evex512.
[1] https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;h=c052a6f4
[2] https://bugs.gentoo.org/956632
Signed-off-by: Kostadin Shishmanov <kostadinshishmanov@protonmail.com>
MSAN has been updated since 2022 when this macro was added
and as such the working around MSAN's deficient understanding
of the fstat/stat syscall behaviour at the time is no longer
required.
As an effective no-op a straight removal is sufficient.
As of CMake 3.24 CMAKE_COMPILER_IS_GNU(CC|CXX) are deprecated and should
be replaced with CMAKE_(C|CXX)_COMPILER_ID which were introduced with
CMake 2.6.
Initialize UErrorCode to U_ZERO_ERROR before passing it to
ucal_getDefaultTimeZone(), as required by the ICU conventions.
Passing an uninitialized status variable leads to undefined behavior and
non-deterministic failures.
This issue was not observed on x64 builds, but appeared during
Windows/ARM64 CI testing. The lack of initialization led to
inconsistent fallback between ICU and native Windows time zone.
mtr tests with restarts showed the alternation of system_time_zone,
as reported by check_testcase after the test.
-SYSTEM_TIME_ZONE Coordinated Universal Time
+SYSTEM_TIME_ZONE Etc/UTC
In commit b6923420f3 (MDEV-29445)
we started to specify the MAP_POPULATE flag for allocating the
InnoDB buffer pool. This would cause a lot of time to be spent
on __mm_populate() inside the Linux kernel, such as 16 seconds
to pre-fault or commit innodb_buffer_pool_size=64G.
Let us revert to the previous way of allocating the buffer pool
at startup. Note: An attempt to increase the buffer pool size by
SET GLOBAL innodb_buffer_pool_size (up to innodb_buffer_pool_size_max)
will invoke my_virtual_mem_commit(), which will use MAP_POPULATE
to zero-fill and prefault the requested additional memory area, blocking
buf_pool.mutex.
Before MDEV-29445 we allocated the InnoDB buffer pool by invoking
mmap(2) once (via my_large_malloc()). After the change, we would
invoke mmap(2) twice, first via my_virtual_mem_reserve() and then
via my_virtual_mem_commit(). Outside Microsoft Windows, we are
reverting back to my_large_malloc() like allocation.
my_virtual_mem_reserve(): Define only for Microsoft Windows.
Other platforms should invoke my_large_virtual_alloc() and
update_malloc_size() instead of my_virtual_mem_reserve() and
my_virtual_mem_commit().
my_large_virtual_alloc(): Define only outside Microsoft Windows.
Do not specify MAP_NORESERVE nor MAP_POPULATE, to preserve compatibility
with my_large_malloc(). Were MAP_POPULATE specified, the mmap()
system call would be significantly slower, for example 18 seconds
to reserve 64 GiB upfront.
Fixing the code adding MySQL _0900_ collations as _uca1400_ aliases
not to perform deep initialization of the corresponding _uca1400_
collations.
Only basic initialization is now performed which allows to watch
these collations (both _0900_ and _uca1400_) in queries to
INFORMATION_SCHEMA tables COLLATIONS and
COLLATION_CHARACTER_SET_APPLICABILITY,
as well as in SHOW COLLATION statements.
Deep initialization is now performed only when a collation
(either the _0900_ alias or the corresponding _uca1400_ collation)
is used for the very first time after the server startup.
Refactoring was done to maintain the code easier:
- most of the _uca1400_ code was moved from ctype-uca.c
to a new file ctype-uca1400.c
- most of the _0900_ code was moved from type-uca.c
to a new file ctype-uca0900.c
Change details:
- The original function add_alias_for_collation() added by the patch for
"MDEV-20912 Add support for utf8mb4_0900_* collations in MariaDB Server"
was removed from mysys/charset.c, as it had two two problems:
a. it forced deep initialization of the _uca1400_ collations
when adding _0900_ aliases for them at the server startup
(the main reported problem)
b. the collation initialization code in add_alias_for_collation()
was related more to collations rather than to memory management,
so /strings should be a better place for it than /mysys.
The code from add_alias_for_collation() was split into separate functions.
Cyclic dependency was removed. `#include <my_sys.h>` was removed
from /strings/ctype-uca.c. Collations are now added using a callback
function MY_CHARSET_LOADED::add_collation, like it is done for
user collations defined in Index.xml. The code in /mysys sets
MY_CHARSET_LOADED::add_collation to add_compiled_collation().
- The function compare_collations() was removed.
A new virtual function was added into my_collation_handler_st instead:
my_bool (*eq_collation)(CHARSET_INFO *self, CHARSET_INFO *other);
because it is the collation handler who knows how to detect equal
collations by comparing only some of CHARSET_INFO members without
their deep initialization.
Three implementations were added:
- my_ci_eq_collation_uca() for UCA collations, it compares
_0900_ collations as equal to their corresponding _uca1400_ collations.
- my_ci_eq_collation_utf8mb4_bin(), it compares
utf8mb4_nopad_bin and utf8mb4_0900_bin as equal.
- my_ci_eq_collation_generic() - the default implementation,
which compares all collations as not equal.
A C++ wrapper CHARSET_INFO::eq_collations() was added.
The code in /sql was changes to use the wrapper instead of
the former calls for the removed function compare_collations().
- A part of add_alias_for_collation() was moved into a new function
my_ci_alloc(). It allocates a memory for a new charset_info_st
instance together with the collation name and the comment using a single
MY_CHARSET_LOADER::once_alloc call, which points to my_once_alloc()
in the server.
- A part of add_alias_for_collation() was moved into a new function
my_ci_make_comment_for_alias(). It makes an "Alias for xxx" string,
e.g. "Alias for utf8mb4_uca1400_swedish_ai_ci" in case of
utf8mb4_sv_0900_ai_ci.
- A part of the code in create_tailoring() was moved to
a new function my_uca1400_collation_get_initialized_shared_uca(),
to reuse the code between _uca1400_ and _0900_ collations.
- A new function my_collation_id_is_mysql_uca0900() was added
in addition to my_collation_id_is_mysql_uca1400().
- Functions to build collation names were added:
my_uca0900_collation_build_name()
my_uca1400_collation_build_name()
- A shared function function was added:
my_bool
my_uca1400_collation_alloc_and_init(MY_CHARSET_LOADER *loader,
LEX_CSTRING name,
LEX_CSTRING comment,
const uca_collation_def_param_t *param,
uint id)
It's reused to add _uca1400_ and _0900_ collations, with basic
initialization (without deep initialization).
- The function add_compiled_collation() changed its return type from
void to int, to make it compatible with MY_CHARSET_LOADER::add_collation.
- Functions mysql_uca0900_collation_definition_add(),
mysql_uca0900_utf8mb4_collation_definitions_add(),
mysql_utf8mb4_0900_bin_add() were added into ctype-uca0900.c.
They get MY_CHARSET_LOADER as a parameter.
- Functions my_uca1400_collation_definition_add(),
my_uca1400_collation_definitions_add() were moved from
charset-def.c to strings/ctype-uca1400.c.
The latter now accepts MY_CHARSET_LOADER as the first parameter
instead of initializing a MY_CHARSET_LOADER inside.
- init_compiled_charsets() now initializes a MY_CHARSET_LOADER
variable and passes it to all functions adding collations:
- mysql_utf8mb4_0900_collation_definitions_add()
- mysql_uca0900_utf8mb4_collation_definitions_add()
- mysql_utf8mb4_0900_bin_add()
- A new structure was added into ctype-uca.h:
typedef struct uca_collation_def_param
{
my_cs_encoding_t cs_id;
uint tailoring_id;
uint nopad_flags;
uint level_flags;
} uca_collation_def_param_t;
It simplifies reusing the code for _uca1400_ and _0900_ collations.
- The definition of MY_UCA1400_COLLATION_DEFINITION was
moved from ctype-uca.c to ctype-uca1400.h, to reuse
the code for _uca1400_ and _0900_ collations.
- The definitions of "MY_UCA_INFO my_uca_v1400" and
"MY_UCA_INFO my_uca1400_info_tailored[][]" were moved from
ctype-uca.c to ctype-uca1400.c.
- The definitions/declarations of:
- mysql_0900_collation_start,
- struct mysql_0900_to_mariadb_1400_mapping
- mysql_0900_to_mariadb_1400_mapping
- mysql_utf8mb4_0900_collation_definitions_add()
were moved from ctype-uca.c to ctype-uca0900.c
- Functions
my_uca1400_make_builtin_collation_id()
my_uca1400_collation_definition_init()
my_uca1400_collation_id_uca400_compat()
my_ci_get_collation_name_uca1400_context()
were moved from ctype-uca.c to ctype-uca1400.c and ctype-uca1400.h
- A part of my_uca1400_collation_definition_init()
was moved into my_uca0520_builtin_collation_by_id(),
to make functions smaller.
