Commit graph

7597 commits

Author SHA1 Message Date
Oleksandr Byelkin
d6444022ca Merge branch 'bb-11.5-release' into bb-11.6-release 2024-08-06 17:28:38 +02:00
Oleksandr Byelkin
ea75a0b600 Merge branch '11.4' into 11.5 2024-08-05 17:50:18 +02:00
Oleksandr Byelkin
1640c9b06e Merge branch '11.2' into 11.4 2024-08-04 17:27:48 +02:00
Oleksandr Byelkin
dced6cbdb6 Merge branch '11.1' into 11.2 2024-08-03 09:50:16 +02:00
Oleksandr Byelkin
80abd847da Merge branch '10.11' into 11.1 2024-08-03 09:32:42 +02:00
Oleksandr Byelkin
0e8fb977b0 Merge branch '10.6' into 10.11 2024-08-03 09:15:40 +02:00
Oleksandr Byelkin
8f020508c8 Merge branch '10.5' into 10.6 2024-08-03 09:04:24 +02:00
Marko Mäkelä
232d7a5e2d MDEV-34565: SIGILL due to OS not supporting AVX512
It is not sufficient to check that the CPU supports the necessary
instructions. Also the operating system (or virtual machine hypervisor)
must enable all the AVX registers to be saved and restored on a
context switch.

Because clang 8 does not support the compiler intrinsic _xgetbv()
we will require clang 9 or later for enabling the use of VPCLMULQDQ
and the related AVX512 features.
2024-07-29 10:58:09 +03:00
Oleksandr Byelkin
0fe39d368a Merge branch '10.6' into 10.11 2024-07-22 15:14:50 +02:00
Oleksandr Byelkin
9af2caca33 Merge branch '10.5' into 10.6 2024-07-18 16:25:33 +02:00
Sergei Golubchik
f12634f5a4 MDEV-34530 dead code in the thr_rwlock.c
remove it
2024-07-17 21:25:40 +02:00
Alexander Barkov
a2a5ba14a8 Merge remote-tracking branch 'origin/11.5' into 11.6 2024-07-10 19:18:52 +04:00
Alexander Barkov
4e805aed85 Merge remote-tracking branch 'origin/11.4' into 11.5 2024-07-10 12:17:09 +04:00
Alexander Barkov
5fb07d942b Merge remote-tracking branch 'origin/11.2' into 11.4 2024-07-09 21:45:37 +04:00
Vladislav Vaintroub
186a1afe63 MDEV-32537 due to Linux, restrict thread name to 15 characters, also in PS.
Rename some threads to workaround this restrictions,
e.g "rpl_parallel_thread"->"rpl_parallel",
"slave_background" -> "slave_bg" etc.
2024-07-09 13:20:49 +02:00
Vladislav Vaintroub
5bd0516488 MDEV-32537 Name threads to improve debugging experience and diagnostics.
Use SetThreadDescription/pthread_setname_np to give threads a name.
2024-07-09 13:17:20 +02:00
Vladislav Vaintroub
584fc85e21 MDEV-32537 Name threads to improve debugging experience and diagnostics.
Use SetThreadDescription/pthread_setname_np to give threads a name.
2024-07-09 13:17:20 +02:00
Alexander Barkov
8aad19ddfc Merge remote-tracking branch 'origin/11.1' into 11.2 2024-07-09 14:04:11 +04:00
Alexander Barkov
44af9bfc67 Merge remote-tracking branch 'origin/10.11' into 11.1 2024-07-09 10:45:47 +04:00
Alexander Barkov
4d71a117a3 Merge remote-tracking branch 'origin/10.6' into 10.11 2024-07-08 21:52:08 +04:00
Alexander Barkov
e56040fee8 Merge remote-tracking branch 'origin/10.5' into 10.6 2024-07-08 18:59:04 +04:00
Alexander Barkov
027f137741 Merge remote-tracking branch 'origin/11.5' into 11.6 2024-07-08 14:15:04 +04:00
Anson Chung
df35072c8d Refactor GitLab cppcheck and update SAST ignorelists
Line numbers had to be removed from the ignorelists in order to be
diffed against since locations of the same findings can differ
across runs. Therefore preprocessing has to be done on the CI findings
so that it can be compared to the ignorelist and new findings can be
outputted. However, since line numbers have to be removed, a situation
occurs where it is difficult to reference the location of findings
in code given the output of the CI job.

To lessen this pain, change the cppcheck template to include
code snippets which make it easier to reference where in the code
the finding is referring to, even in the absence of line numbers.
Ignorelisting works as before since locations of the finding may
change but not the code it is referring to.

Furthermore, due to the innate difficulty in maintaining ignorelists
across branches and triaging new findings, allow failure as to not
have constantly failing pipelines as a result of a new findings that
have not been addressed yet.

Lastly, update SAST ignorelists to match the newly refactored cppcheck
job and the current state of the codebase.

All new code of the whole pull request, including one or several
files that are either new files or modified ones, are contributed
under the BSD-new license. I am contributing on behalf of my
employer Amazon Web Services, Inc.
2024-07-08 10:51:48 +01:00
Anson Chung
215fab68db Perform simple fixes for cppcheck findings
Rectify cases of mismatched brackets and address
possible cases of division by zero by checking if
the denominator is zero before dividing.

No functional changes were made.

All new code of the whole pull request, including one or several
files that are either new files or modified ones, are contributed
under the BSD-new license. I am contributing on behalf of my
employer Amazon Web Services, Inc.
2024-07-08 10:51:48 +01:00
Alexander Barkov
8f4ec79d09 Merge remote-tracking branch 'origin/11.4' into 11.5 2024-07-08 12:25:04 +04:00
Marko Mäkelä
72ceae7314 MDEV-34510: UBSAN: overflow on adding an unsigned offset
crc32_avx512(): Explicitly cast ssize_t(size) to make it clear that
we are indeed applying a negative offset to a pointer.
2024-07-08 18:17:29 +10:00
Hugo Wen
9e8546e2bd Fix a stack overflow in pinbox allocator
MariaDB supports a "wait-free concurrent allocator based on pinning addresses".
In `lf_pinbox_real_free()` it tries to sort the pinned addresses for better
performance to use binary search during "real free". `alloca()` was used to
allocate stack memory and copy addresses.

To prevent a stack overflow when allocating the stack memory the function checks
if there's enough stack space. However, the available stack size was calculated
inaccurately which eventually caused database crash due to stack overflow.

The crash was seen on MariaDB 10.6.11 but the same code defect exists on all
MariaDB versions.

A similar issue happened previously and the fix in fc2c1e43 was to add a
`ALLOCA_SAFETY_MARGIN` which is 8192 bytes. However, that safety margin is not
enough during high connection workloads.

MySQL also had a similar issue and the fix
https://github.com/mysql/mysql-server/commit/b086fda was to remove the use of
`alloca` and replace qsort approach by a linear scan through all pointers (pins)
owned by each thread.

This commit is mostly the same as it is the only way to solve this issue as:
1. Frame sizes in different architecture can be different.
2. Number of active (non-null) pinned addresses varies, so the frame
   size for the recursive sorting function `msort_with_tmp` is also hard
   to predict.
3. Allocating big memory blocks in stack doesn't seem to be a very good
   practice.

For further details see the mentioned commit in MySQL and the inline comments.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
2024-07-05 13:26:37 +10:00
Monty
d8c9c5ead6 MDEV-34491 Setting log_slow_admin="" at startup should be converted to log_slow_admin=ALL
We have an issue if a user have the following in a configuration file:
log_slow_filter=""                  # Log everything to slow query log
log_queries_not_using_indexes=ON

This set log_slow_filter to 'not_using_index' which disables
slow_query_logging of most queries.
In effect, on should never use log_slow_filter="" in config files but
instead use log_slow_filter=ALL.

Fixed by changing log_slow_filter="" that comes either from a
configuration file or from the command line, when starting to the server,
to log_slow_filter=ALL.
A warning will be printed when this happens.

Other things:
- One can now use =ALL for any 'set' variable to set all options at once.
  (backported from 10.6)
2024-07-02 11:26:13 +03:00
Vladislav Vaintroub
7c5fdc9b6a MDEV-33748 get rid of pthread_(get_/set_)specific, use thread_local
Apart from better performance when accessing thread local variables,
we'll get rid of things that depend on initialization/cleanup of
pthread_key_t variables.

Where appropriate, use compiler-dependent pre-C++11 thread-local
equivalents, where it makes sense, to avoid initialization check overhead
that non-static thread_local can suffer from.
2024-06-21 13:46:41 +02:00
Alexander Barkov
c4bf4ce948 Merge remote-tracking branch 'origin/11.2' into 11.4 2024-06-17 15:46:39 +04:00
Marko Mäkelä
a21e49cbcc Merge 11.1 into 11.2 2024-06-17 12:02:03 +03:00
Marko Mäkelä
d34289a3e2 Merge 10.11 into 11.1 2024-06-17 09:21:50 +03:00
Marko Mäkelä
b81d717387 Merge 10.6 into 10.11 2024-06-11 12:50:10 +03:00
Marko Mäkelä
27834ebc91 Merge 10.5 into 10.6 2024-06-10 15:22:15 +03:00
Marko Mäkelä
a2bd936c52 MDEV-33161 Function pointer signature mismatch in LF_HASH
In cmake -DWITH_UBSAN=ON builds with clang but not with GCC,
-fsanitize=undefined will flag several runtime errors on
function pointer mismatch related to the lock-free hash table LF_HASH.

Let us use matching function signatures and remove function pointer
casts in order to avoid potential bugs due to undefined behaviour.

These errors could be caught at compilation time by
-Wcast-function-type-strict, which is available starting with clang-16,
but not available in any version of GCC as of now. The old GCC flag
-Wcast-function-type is enabled as part of -Wextra, but it specifically
does not catch these errors.

Reviewed by: Vladislav Vaintroub
2024-06-10 12:35:33 +03:00
Marko Mäkelä
d9d0e8fd60 MDEV-34321: call to crc32c_3way through pointer to incorrect function type
In commit 9ec7819c58 the CRC-32 function
signatures had been unified somewhat, but not enough.

clang -fsanitize=undefined would flag a function pointer signature
mismatch between const char* and const void*, but not between
uint32_t and unsigned. We try to fix both inconsistencies anyway.

Reviewed by: Vladislav Vaintroub
2024-06-07 13:51:46 +03:00
Yuchen Pei
2d3e2c58b6
Merge branch '10.11' into 11.1 2024-05-31 10:54:31 +10:00
Marko Mäkelä
22ba7e4ff8 Merge 10.6 into 10.11 2024-05-30 16:04:00 +03:00
Marko Mäkelä
5ba542e9ee Merge 10.5 into 10.6 2024-05-30 14:27:07 +03:00
Sergei Golubchik
bfbdc273d2 my_errno can not be set to EE_ error numbers
EE_ numbers occupy the same range as OS Exxx errors and
my_errno is generally for OS errors. And for HA_ERR_ handler errors
which occupy a different range, so can be freely mixed with OS errors.
2024-05-27 12:39:04 +02:00
Sergei Golubchik
ce5b8e5944 cleanup: clarify the function name and purpose
and make it static
2024-05-27 12:39:04 +02:00
Monty
b9f5793176 MDEV-9101 Limit size of created disk temporary files and tables
Two new variables added:
- max_tmp_space_usage : Limits the the temporary space allowance per user
- max_total_tmp_space_usage: Limits the temporary space allowance for
  all users.

New status variables: tmp_space_used & max_tmp_space_used
New field in information_schema.process_list: TMP_SPACE_USED

The temporary space is counted for:
- All SQL level temporary files. This includes files for filesort,
  transaction temporary space, analyze, binlog_stmt_cache etc.
  It does not include engine internal temporary files used for repair,
  alter table, index pre sorting etc.
- All internal on disk temporary tables created as part of resolving a
  SELECT, multi-source update etc.

Special cases:
- When doing a commit, the last flush of the binlog_stmt_cache
  will not cause an error even if the temporary space limit is exceeded.
  This is to avoid giving errors on commit. This means that a user
  can temporary go over the limit with up to binlog_stmt_cache_size.

Noteworthy issue:
- One has to be careful when using small values for max_tmp_space_limit
  together with binary logging and with non transactional tables.
  If a the binary log entry for the query is bigger than
  binlog_stmt_cache_size and one hits the limit of max_tmp_space_limit
  when flushing the entry to disk, the query will abort and the
  binary log will not contain the last changes to the table.
  This will also stop the slave!
  This is also true for all Aria tables as Aria cannot do rollback
  (except in case of crashes)!
  One way to avoid it is to use @@binlog_format=statement for
  queries that updates a lot of rows.

Implementation:
- All writes to temporary files or internal temporary tables, that
  increases the file size, are routed through temp_file_size_cb_func()
  which updates and checks the temp space usage.
- Most of the temporary file monitoring is done inside IO_CACHE.
  Temporary file monitoring is done inside the Aria engine.
- MY_TRACK and MY_TRACK_WITH_LIMIT are new flags for ini_io_cache().
  MY_TRACK means that we track the file usage. TRACK_WITH_LIMIT means
  that we track the file usage and we give an error if the limit is
  breached. This is used to not give an error on commit when
  binlog_stmp_cache is flushed.
- global_tmp_space_used contains the total tmp space used so far.
  This is needed quickly check against max_total_tmp_space_usage.
- Temporary space errors are using EE_LOCAL_TMP_SPACE_FULL and
  handler errors are using HA_ERR_LOCAL_TMP_SPACE_FULL.
  This is needed until we move general errors to it's own error space
  so that they cannot conflict with system error numbers.
- Return value of my_chsize() and mysql_file_chsize() has changed
  so that -1 is returned in the case my_chsize() could not decrease
  the file size (very unlikely and will not happen on modern systems).
  All calls to _chsize() are updated to check for > 0 as the error
  condition.
- At the destruction of THD we check that THD::tmp_file_space == 0
- At server end we check that global_tmp_space_used == 0
- As a precaution against errors in the tmp_space_used code, one can set
  max_tmp_space_usage and max_total_tmp_space_usage to 0 to disable
  the tmp space quota errors.
- truncate_io_cache() function added.
- Aria tables using static or dynamic row length are registered in 8K
  increments to avoid some calls to update_tmp_file_size().

Other things:
- Ensure that all handler errors are registered.  Before, some engine
  errors could be printed as "Unknown error".
- Fixed bug in filesort() that causes a assert if there was an error
  when writing to the temporay file.
- Fixed that compute_window_func() now takes into account write errors.
- In case of parallel replication, rpl_group_info::cleanup_context()
  could call trans_rollback() with thd->error set, which would cause
  an assert. Fixed by resetting the error before calling trans_rollback().
- Fixed bug in subselect3.inc which caused following test to use
  heap tables with low value for max_heap_table_size
- Fixed bug in sql_expression_cache where it did not overflow
  heap table to Aria table.
- Added Max_tmp_disk_space_used to slow query log.
- Fixed some bugs in log_slow_innodb.test
2024-05-27 12:39:04 +02:00
Monty
775cba4d0f MDEV-33145 Add FLUSH GLOBAL STATUS
- FLUSH GLOBAL STATUS now resets most global_status_vars.
  At this stage, this is mainly to be used for testing.
- FLUSH SESSION STATUS added as an alias for FLUSH STATUS.
- FLUSH STATUS does not require any privilege (before required RELOAD).
- FLUSH GLOBAL STATUS requires RELOAD privilege.
- All global status reset moved to FLUSH GLOBAL STATUS.
- Replication semisync status variables are now reset by
  FLUSH GLOBAL STATUS.
- In test cases, the only changes are:
  - Replace FLUSH STATUS with FLUSH GLOBAL STATUS
  - Replace FLUSH STATUS with FLUSH STATUS; FLUSH GLOBAL STATUS.
    This was only done in a few tests where the test was using SHOW STATUS
    for both local and global variables.
- Uptime_since_flush_status is now always provided, independent if
  ENABLED_PROFILING is enabled when compiling MariaDB.
- @@global.Uptime_since_flush_status is reset on FLUSH GLOBAL STATUS
  and @@session.Uptime_since_flush_status is reset on FLUSH SESSION STATUS.
- When connected, @@session.Uptime_since_flush_status is set to 0.
2024-05-27 12:39:03 +02:00
Anson Chung
dcbc526a20 Alter thr_timer to allow server startup past 2038
In previous commits, changes were made to use the entire 32 bit
unsigned range in order to support timestamps past 2038 on
64 bit systems. Without changing set_max_time on thr_timer to also
match this new range, the server will crash when attempting to
startup past 2038-01-19. Like the previous commits, this only
applies to 64 bit systems.

All new code of the whole pull request, including one or several
files that are either new files or modified ones, are contributed
under the BSD-new license. I am contributing on behalf of my
employer Amazon Web Services, Inc.
2024-05-27 12:39:03 +02:00
Monty
6254fcf84a Fix that one cannot change value for deleted variables
This enasures that the 'variables that only exists for compatibility'
are always kept as its default value.

Based on an idea from Sergei Golubchik
2024-05-27 12:39:02 +02:00
Sergei Golubchik
df10a945fc MDEV-28671 post-merge fixes
* use new deprecated printer for all deprecated server options
* restore alphabetic option sorting order
* move deprecated printer from mysqld.cc to my_getopt.c
* in --help print deprecation message at the end of the option help
* move 'ALL' help text where it belongs - to other SET options, and
  with a correct indentation.
* consistently end all or none command-line option help strings
  with a dot - my_print_help() needs that.
  It's about 50/50 now, so let's do none, less line wraps in --help
* remove trailing spaces from command-line option help strings
2024-05-27 12:39:02 +02:00
Christian Gonzalez
4186fa72fb MDEV-28671 Enable var deprecation for mysqld help output
Currently there are mechanism to mark a system variable as
deprecated, but they are only used to print warning messages
when a deprecated variable is set.

Leverage the existing mechanisms in order to make the
deprecation information available at the --help output of mysqld by:

* Moving the deprecation information (i.e `deprecation_substitute`
  attribute) from the `sys_var` class into the `my_option` struct.
  As every `sys_var` contains its own `my_option` struct, the access
  to the deprecation information remains available to `sys_var`
  objects. `my_getotp` functions, which works directly with
  `my_option` structs, gain access to this information while building
  the --help output.

* For plugin variables, leverages the `PLUGIN_VAR_DEPRECATED` flag
  and set the `deprecation_substitute` attribute  accordingly when
  building the `my_option` objects.

* Change the `option_cmp` function to use the `deprecation_substitute`
  attribute instead of the name when sorting the options. This way
  deprecated options and the substitutes will be grouped together.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.
2024-05-27 12:39:02 +02:00
Oleksandr Byelkin
dd7d9d7fb1 Merge branch '11.4' into 11.5 2024-05-23 17:01:43 +02:00
Oleksandr Byelkin
99b370e023 Merge branch '11.2' into 11.4 2024-05-21 19:38:51 +02:00
Marko Mäkelä
266495b93e MDEV-33817 fixup: Disable for macOS
According to https://discussions.apple.com/thread/8256853
an attempt to use AVX512 registers on macOS will result in #UD
(crash at runtime).

Also, starting with clang-18 and GCC 14, we must add "evex512" to the
target flags so that AVX and SSE instructions can use AVX512 specific
encodings. This flag was introduced together with the avx10.1-512 target.
Older compiler versions do not recognize "evex512". We do not want to
write "avx10.1-512" because it could enable some AVX512 subfeatures
that we do not have any CPUID check for.

Reviewed by: Vladislav Vaintroub
Tested on macOS by: Valerii Kravchuk
2024-05-21 16:45:07 +03:00