Commit graph

7651 commits

Author SHA1 Message Date
Marko Mäkelä
6680fd8d4b Merge 10.6 into 10.7 2022-06-21 18:02:41 +03:00
Marko Mäkelä
3794673111 MDEV-28836: Memory alignment cleanup
Table_cache_instance: Define the structure aligned at
the CPU cache line, and remove a pad[] data member.
Krunal Bauskar reported this to improve performance on ARMv8.

aligned_malloc(): Wrapper for the Microsoft _aligned_malloc()
and the ISO/IEC 9899:2011 <stdlib.h> aligned_alloc().
Note: The parameters are in the Microsoft order (size, alignment),
opposite of aligned_alloc(alignment, size).
Note: The standard defines that size must be an integer multiple
of alignment. It is enforced by AddressSanitizer but not by GNU libc
on Linux.

aligned_free(): Wrapper for the Microsoft _aligned_free() and
the standard free().

HAVE_ALIGNED_ALLOC: A new test. Unfortunately, support for
aligned_alloc() may still be missing on some platforms.
We will fall back to posix_memalign() for those cases.

HAVE_MEMALIGN: Remove, along with any use of the nonstandard memalign().

PFS_ALIGNEMENT (sic): Removed; we will use CPU_LEVEL1_DCACHE_LINESIZE.

PFS_ALIGNED: Defined using the C++11 keyword alignas.

buf_pool_t::page_hash_table::create(),
lock_sys_t::hash_table::create():
lock_sys_t::hash_table::resize(): Pad the allocation size to an
integer multiple of the alignment.

Reviewed by: Vladislav Vaintroub
2022-06-21 16:59:49 +03:00
Marko Mäkelä
0af9346079 Merge 10.7 into 10.8 2022-06-09 14:37:53 +03:00
Marko Mäkelä
fe75e5e5b1 Merge 10.6 into 10.7 2022-06-09 14:11:43 +03:00
Marko Mäkelä
e11b82f8f5 Merge 10.5 into 10.6 2022-06-09 13:34:52 +03:00
Daniel Black
e8b0894dc8 MDEV-28243: AIX missing my_gethwaddr implementation
and failing spider partition test.

With some small datatype changes to the Linux/Solaris my_gethwaddr implementation
the hardware address of AIX can be returned. This is an important aspect
in Spider (and UUID).

Spider test change reviewed by Nayuta Yanagisawa.

my_gethwaddr review by Monty in 
2022-06-08 17:13:51 +10:00
Marko Mäkelä
133c2129cd Merge 10.7 into 10.8 2022-04-27 10:43:00 +03:00
Marko Mäkelä
638afc4acf Merge 10.6 into 10.7 2022-04-26 18:59:40 +03:00
Marko Mäkelä
e135edec3a Merge 10.5 into 10.6 2022-04-26 15:21:20 +03:00
Marko Mäkelä
4faef6e240 Cleanup: Remove IF_VALGRIND
The purpose of the compress() wrapper my_compress_buffer() was twofold:
silence Valgrind warnings about uninitialized memory access before
zlib 1.2.4, and have PERFORMANCE_SCHEMA instrumentation of some zlib
related memory allocation. Because of PERFORMANCE_SCHEMA, we cannot
trivially replace my_compress_buffer() with compress().

az_open(): Remove a crc32() call. Any CRC of the empty string is 0.
2022-04-25 09:40:40 +03:00
Marko Mäkelä
5c69e93630 Merge 10.7 into 10.8 2022-03-30 09:34:07 +03:00
Marko Mäkelä
a4d753758f Merge 10.6 into 10.7 2022-03-30 08:52:05 +03:00
Marko Mäkelä
b242c3141f Merge 10.5 into 10.6 2022-03-29 16:16:21 +03:00
Marko Mäkelä
d62b0368ca Merge 10.4 into 10.5 2022-03-29 12:59:18 +03:00
Marko Mäkelä
ae6e214fd8 Merge 10.3 into 10.4 2022-03-29 11:13:18 +03:00
Vladislav Vaintroub
739002eec9 MDEV-28178 Windows : sporadic ER_ERROR_ON_RENAME .. (errno: 13 "Permission denied")
On affected machine, the error happens sporadically in
innodb.instant_alter_limit.

Procmon shows SetRenameInformationFile failing with ERROR_ACCESS_DENIED.
In this case, the destination file was previously opened rsp oplocked by
Windows defender antivirus.

The fix is to retry MoveFileEx on ERROR_ACCESS_DENIED.
2022-03-28 12:46:35 +02:00
Marko Mäkelä
18bb95b608 Merge 10.7 into 10.8 2022-03-14 11:52:11 +02:00
Marko Mäkelä
e67d46e4a1 Merge 10.6 into 10.7 2022-03-14 11:30:32 +02:00
Marko Mäkelä
572e34304e Merge 10.5 into 10.6 2022-03-14 10:59:46 +02:00
Marko Mäkelä
c2146ce774 MDEV-24841: More workarounds
For some reason, the tests of the MemorySanitizer build on 10.5 failed
with both clang 13 and clang 14 with SIGSEGV. On 10.6 where it worked
better, some more places to work around were identified.
2022-03-14 10:37:39 +02:00
Marko Mäkelä
59359fb44a MDEV-24841 Build error with MSAN use-of-uninitialized-value in comp_err
The MemorySanitizer implementation in clang includes some built-in
instrumentation (interceptors) for GNU libc. In GNU libc 2.33, the
interface to the stat() family of functions was changed. Until the
MemorySanitizer interceptors are adjusted, any MSAN code builds
will act as if that the stat() family of functions failed to initialize
the struct stat.

A fix was applied in
https://reviews.llvm.org/rG4e1a6c07052b466a2a1cd0c3ff150e4e89a6d87a
but it fails to cover the 64-bit variants of the calls.

For now, let us work around the MemorySanitizer bug by defining
and using the macro MSAN_STAT_WORKAROUND().
2022-03-14 09:28:55 +02:00
Marko Mäkelä
32d741b5b0 Merge 10.7 into 10.8 2022-02-25 16:24:13 +02:00
Marko Mäkelä
3d88f9f34c Merge 10.6 into 10.7 2022-02-25 16:09:16 +02:00
Marko Mäkelä
e04b5eaa79 Merge 10.5 into 10.6 2022-02-25 12:01:21 +02:00
Like Ma
97ed3dd6df Remove unused header from crc32c.cc 2022-02-24 19:41:00 +11:00
Marko Mäkelä
c75e3770dc Merge 10.7 into 10.8 2022-02-09 16:24:19 +02:00
Marko Mäkelä
70a8875564 Merge 10.6 into 10.7 2022-02-09 16:04:49 +02:00
Marko Mäkelä
cce994057b Merge 10.5 into 10.6 2022-02-09 15:49:50 +02:00
Monty
88fb89acb7 Fixes some compiler issues on AIX ( 2022-02-08 14:32:28 +02:00
Monty
df02de68f3 Fixed my_addr_resolve (cherry picked from 10.6)
When a server is compiled with -fPIE, my_addr_resolve needs to
subtract the info.dli_fbase from symbol addresses in memory for
addr2line to recognize them.  When a server is compiled without -fPIE,
my_addr_resolve should not do it.  Unfortunately not all compilers
define __PIE__ when -fPIE was used (e.g. older gcc doesn't), so we
have to resort to run-time detection.
2022-02-08 14:32:28 +02:00
Marko Mäkelä
5b3ad94c7b MDEV-27208: Extend CRC32() and implement CRC32C()
We used to define a native unary function CRC32() that computes the CRC-32
of a string using the ISO 3309 polynomial that is being used by zlib
and many others.

Often, a CRC is computed in pieces. To faciliate this, we introduce a
2-ary variant of the function that inputs a previous CRC as the first
argument: CRC32('MariaDB')=CRC32(CRC32('Maria'),'DB').

InnoDB and MyRocks use a different polynomial, which was implemented
in SSE4.2 instructions that were introduced in the
Intel Nehalem microarchitecture. This is commonly called CRC-32C
(Castagnoli).

We introduce a native function that uses the Castagnoli polynomial:
CRC32C('MariaDB')=CRC32C(CRC32C('Maria'),'DB'). This allows
SELECT...INTO DUMPFILE to be used for the creation of files with
valid checksums, such as a logically empty InnoDB redo log file
ib_logfile0 corresponding to a particular log sequence number.
2022-01-21 19:24:00 +02:00
Vladislav Vaintroub
e222e44d1b Merge branch 'preview-10.8-MDEV-26713-Windows-i18-support' into 10.8 2022-01-18 21:37:52 +01:00
Vladislav Vaintroub
2e48fbe3f5 MDEV-27525 Invalid (non-UTF8) characters found for option 'plugin_dir'
Two Problems
1. Upgrade wizard failed to retrieve path to service executable,
if it contained non-ASCII.
Fixed by setlocale(LC_ALL, "en_US.UTF8"), which was missing in upgrade wizard

2.mysql_upgrade_service only updated (converted to UTF8) the server's sections
leaving client's as-is

Corrected typo.

3. Fixed assertion in my_getopt, turns out to be too strict.
2022-01-18 17:32:53 +01:00
Daniel Black
d434250ee1 MDEV-25342: autosize innodb_buffer_pool_chunk_size
The previous default innodb_buffer_pool_chunk_size of 128M
made sense when the innodb buffer pool size was a few GB.

When the pool size is 128GB this means the chunk size is 0.1%
of this. Fine tuning the buffer pool size on such a fine
increment doesn't make practical sense. Also on extremely
large buffer pool systems, initializing on the default 128M can
also take a considerable amount of time.

When large pages are enabled, the chunk size has to be a multiple
of an available large page size or memory allocation without
use can occur.

Previously the default 0 was documented as disabling resizing.
With srv_buf_pool_chunk_unit > 0 assertions in the code and the
minimium value set, I doubt this was ever the case.

As such the autosizing (based on default 0) takes place as follows:
* a 64th of the innodb_buffer_pool_size
* if large pages, this is rounded down the the nearest multiple
  of the large page size.
* If less than 1MB, set to 1MB.

This does mean the new default innodb_buffer_pool_chunk size is
2MB, derived form the above formular with 128MB as the buffer pool
size.

The innodb_buffer_pool_chunk_size is changed to a size_t for
better compatiblity with the memory allocations which use size_t.
The previous upper limit is changed to the maxium of a size_t. The
maximium value used is the buffer pool size anyway.

Getting this default value of the chunk size to a more practical
size facilitates further development of more automated resizing
without significant overhead or memory fragmentation.

innodb_buffer_pool_resize test adjusted based on 1M default
chunk size thanks Wlad.
2022-01-18 14:20:57 +02:00
alexfanqi
d18f6f2631 MDEV-27429: Support RISC-V cycle timer
Adapted from https://github.com/google/benchmark/pull/833
authored by Sam Elliot at lowRISC.

This requires the RISCV kernel to set the CY bit of the mcountern register
which is done on Linux, but documenting here in case another OS hits
a SIGILL here.

When CY bit of the mcounteren register is unset, reading the cycle register
will cause illegal instruction exception in the next privilege level ( user
mode or supervisor mode ). See the privileged isa manual section 3.1.11 in
https://github.com/riscv/riscv-isa-manual/releases/latest
2022-01-06 10:36:55 +11:00
Julius Goryavsky
3376668ca8 Merge branch 10.2 into 10.3 2021-12-23 14:14:04 +01:00
Daniel Black
4eec6b99e1 MDEV-23175: my_timer_milliseconds clock_gettime for multiple platfomrs
Small postfix to MDEV-23175 to ensure faster option on FreeBSD
and compatibility to Solaris that isn't high resolution.

ftime is left as a backup in case an implementation doesn't
contain any of these clocks.

FreeBSD
    $ ./unittest/mysys/my_rdtsc-t
    1..11
    # ----- Routine ---------------
    # myt.cycles.routine          :             5
    # myt.nanoseconds.routine     :            11
    # myt.microseconds.routine    :            13
    # myt.milliseconds.routine    :            11
    # myt.ticks.routine           :            17
    # ----- Frequency -------------
    # myt.cycles.frequency        :    3610295566
    # myt.nanoseconds.frequency   :    1000000000
    # myt.microseconds.frequency  :       1000000
    # myt.milliseconds.frequency  :           899
    # myt.ticks.frequency         :           136
    # ----- Resolution ------------
    # myt.cycles.resolution       :             1
    # myt.nanoseconds.resolution  :             1
    # myt.microseconds.resolution :             1
    # myt.milliseconds.resolution :             7
    # myt.ticks.resolution        :             1
    # ----- Overhead --------------
    # myt.cycles.overhead         :            26
    # myt.nanoseconds.overhead    :         19140
    # myt.microseconds.overhead   :         19036
    # myt.milliseconds.overhead   :           578
    # myt.ticks.overhead          :         21544
    ok 1 - my_timer_init() did not crash
    ok 2 - The cycle timer is strictly increasing
    ok 3 - The cycle timer is implemented
    ok 4 - The nanosecond timer is increasing
    ok 5 - The nanosecond timer is implemented
    ok 6 - The microsecond timer is increasing
    ok 7 - The microsecond timer is implemented
    ok 8 - The millisecond timer is increasing
    ok 9 - The millisecond timer is implemented
    ok 10 - The tick timer is increasing
    ok 11 - The tick timer is implemented
2021-12-22 16:51:22 +01:00
Daniel Black
12087d6757 MDEV-23175: my_timer_milliseconds clock_gettime for multiple platfomrs
Small postfix to MDEV-23175 to ensure faster option on FreeBSD
and compatibility to Solaris that isn't high resolution.

ftime is left as a backup in case an implementation doesn't
contain any of these clocks.

FreeBSD
    $ ./unittest/mysys/my_rdtsc-t
    1..11
    # ----- Routine ---------------
    # myt.cycles.routine          :             5
    # myt.nanoseconds.routine     :            11
    # myt.microseconds.routine    :            13
    # myt.milliseconds.routine    :            11
    # myt.ticks.routine           :            17
    # ----- Frequency -------------
    # myt.cycles.frequency        :    3610295566
    # myt.nanoseconds.frequency   :    1000000000
    # myt.microseconds.frequency  :       1000000
    # myt.milliseconds.frequency  :           899
    # myt.ticks.frequency         :           136
    # ----- Resolution ------------
    # myt.cycles.resolution       :             1
    # myt.nanoseconds.resolution  :             1
    # myt.microseconds.resolution :             1
    # myt.milliseconds.resolution :             7
    # myt.ticks.resolution        :             1
    # ----- Overhead --------------
    # myt.cycles.overhead         :            26
    # myt.nanoseconds.overhead    :         19140
    # myt.microseconds.overhead   :         19036
    # myt.milliseconds.overhead   :           578
    # myt.ticks.overhead          :         21544
    ok 1 - my_timer_init() did not crash
    ok 2 - The cycle timer is strictly increasing
    ok 3 - The cycle timer is implemented
    ok 4 - The nanosecond timer is increasing
    ok 5 - The nanosecond timer is implemented
    ok 6 - The microsecond timer is increasing
    ok 7 - The microsecond timer is implemented
    ok 8 - The millisecond timer is increasing
    ok 9 - The millisecond timer is implemented
    ok 10 - The tick timer is increasing
    ok 11 - The tick timer is implemented
2021-12-22 16:48:34 +01:00
Vladislav Vaintroub
9ea83f7fbd MDEV-26713 set console codepage to what user set in --default-character-set
If someone on whatever reasons uses --default-character-set=cp850,
this will avoid incorrect display, and inserting incorrect data.

Adjusting console codepage sometimes also needs to happen with
--default-charset=auto, on older Windows. This is because autodetection
is not always exact. For example, console codepage on US editions of
Windows is 437. Client autodetects it as cp850, a rather loose
approximation, given 46 code point differences. We change the console
codepage to cp850, so that there is no discrepancy.

That fix is currently Windows-only, and serves people who used combination
of chcp to achieve WYSIWYG effect (although, this would mostly likely used
with utf8 in the past)

Now, --default-character-set would be a replacement for that.

Fix fs_character_set() detection of current codepage.
2021-12-15 19:13:57 +01:00
Vladislav Vaintroub
a4fc41b6b4 MDEV-26713 Treat codepage 65001 as utf8mb4, not utf8mb3
Also, fix the "UTF8" option in MSI, which is responsible for character-set-server
setting
2021-12-15 19:13:57 +01:00
Vladislav Vaintroub
ba9d231b5a MDEV-26713 Set activeCodePage=UTF8 for windows programs
- Use corresponding entry in the manifest, as described in
https://docs.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page

- If if ANSI codepage is UTF8 (i.e for Windows 1903 and later)
  Use UTF8 as default client charset
  Set console codepage(s) to UTF8, in case process is using console

- Allow some previously disabled MTR tests, that used Unicode for in "exec",
  for the recent Windows versions
2021-12-15 19:13:57 +01:00
Vladislav Vaintroub
99e5ae3b1a MDEV-27090 Windows client - ReadConsoleA does not work correctly with UTF8 codepage
Corresponding Windows bug  https://github.com/microsoft/terminal/issues/4551

Use ReadConsoleW instead and convert to console's input codepage, to
workaround.

Also, disable VT sequences in the console output, as we do not knows what
type of data comes with SELECT, we do not want VT escapes there.

Remove my_cgets()
2021-12-15 19:13:57 +01:00
Vladislav Vaintroub
9e9b211f22 MDEV-27089 Windows : incorrect handling of non-ASCIIs in get_tty_password
Prior to patch, get_password would echo multple mask characters '*', for
a single multibyte input character.

Fixed the behavior by using "wide" version of getch, _getwch.
Also take care of possible characters outside of BMP (i.e we do not print
'*' for high surrogates).

The function will now internally construct the "wide" password string,
and conver to the console codepage. Some characters could still be lost
in that conversion, unless the codepage is utf8, but this is not any new
bug.
2021-12-15 19:13:57 +01:00
Sergei Golubchik
e8a91c18ea Merge branch '10.3' into 10.4 2021-12-07 09:47:42 +01:00
Sergei Golubchik
153b75b576 Merge branch '10.2' into 10.3 2021-12-06 22:23:07 +01:00
Marko Mäkelä
467c7b2b24 Merge 10.7 into 10.8 2021-12-04 13:43:52 +02:00
Marko Mäkelä
182bf9b333 Merge 10.6 into 10.7 2021-12-04 13:23:14 +02:00
Marko Mäkelä
3f7040fa97 Merge 10.5 into 10.6 2021-12-04 12:42:29 +02:00
Martin Beck
4ebac0fc86 MDEV-27088: Server crash on ARM (WMM architecture) due to missing barriers in lf-hash (10.5)
MariaDB server crashes on ARM (weak memory model architecture) while
concurrently executing l_find to load node->key and add_to_purgatory
to store node->key = NULL. l_find then uses key (which is NULL), to
pass it to a comparison function.

The specific problem is the out-of-order execution that happens on a
weak memory model architecture. Two essential reorderings are possible,
which need to be prevented.

a) As l_find has no barriers in place between the optimistic read of
the key field lf_hash.cc#L117 and the verification of link lf_hash.cc#L124,
the processor can reorder the load to happen after the while-loop.

In that case, a concurrent thread executing add_to_purgatory on the same
node can be scheduled to store NULL at the key field lf_alloc-pin.c#L253
before key is loaded in l_find.

b) A node is marked as deleted by a CAS in l_delete lf_hash.cc#L247 and
taken off the list with an upfollowing CAS lf_hash.cc#L252. Only if both
CAS succeed, the key field is written to by add_to_purgatory. However,
due to a missing barrier, the relaxed store of key lf_alloc-pin.c#L253
can be moved ahead of the two CAS operations, which makes the value of
the local purgatory list stored by add_to_purgatory visible to all threads
operating on the list. As the node is not marked as deleted yet, the
same error occurs in l_find.

This change three accesses to be atomic.

* optimistic read of key in l_find lf_hash.cc#L117
* read of link for verification lf_hash.cc#L124
* write of key in add_to_purgatory lf_alloc-pin.c#L253

Reviewers: Sergei Vojtovich, Sergei Golubchik

Fixes: MDEV-23510 / d30c1331a18d875e553f3fcf544997e4f33fb943
2021-11-30 15:16:16 +11:00
Martin Beck
4e0dcf1083 MDEV-27088: Server crash on ARM (WMM architecture) due to missing barriers in lf-hash
MariaDB server crashes on ARM (weak memory model architecture) while
concurrently executing l_find to load node->key and add_to_purgatory
to store node->key = NULL. l_find then uses key (which is NULL), to
pass it to a comparison function.

The specific problem is the out-of-order execution that happens on a
weak memory model architecture. Two essential reorderings are possible,
which need to be prevented.

a) As l_find has no barriers in place between the optimistic read of
the key field lf_hash.cc#L117 and the verification of link lf_hash.cc#L124,
the processor can reorder the load to happen after the while-loop.

In that case, a concurrent thread executing add_to_purgatory on the same
node can be scheduled to store NULL at the key field lf_alloc-pin.c#L253
before key is loaded in l_find.

b) A node is marked as deleted by a CAS in l_delete lf_hash.cc#L247 and
taken off the list with an upfollowing CAS lf_hash.cc#L252. Only if both
CAS succeed, the key field is written to by add_to_purgatory. However,
due to a missing barrier, the relaxed store of key lf_alloc-pin.c#L253
can be moved ahead of the two CAS operations, which makes the value of
the local purgatory list stored by add_to_purgatory visible to all threads
operating on the list. As the node is not marked as deleted yet, the
same error occurs in l_find.

This change three accesses to be atomic.

* optimistic read of key in l_find lf_hash.cc#L117
* read of link for verification lf_hash.cc#L124
* write of key in add_to_purgatory lf_alloc-pin.c#L253

Reviewers: Sergei Vojtovich, Sergei Golubchik

Fixes: MDEV-23510 / d30c1331a18d875e553f3fcf544997e4f33fb943
2021-11-30 11:30:52 +11:00