Commit graph

189820 commits

Author SHA1 Message Date
Alexander Barkov
06fb78c6ac MDEV-21995 Server crashes in Item_field::real_type_handler with table value constructor
1. Code simplification:

Item_default_value handled all these values:
a. DEFAULT(field)
b. DEFAULT
c. IGNORE
and had various conditions to distinguish (a) from (b) and from (c).

Introducing a new abstract class Item_contextually_typed_value_specification,
to handle (b) and (c), so the hierarchy now looks as follows:

Item
  Item_result_field
    Item_ident
      Item_field
        Item_default_value                      - DEFAULT(field)
  Item_contextually_typed_value_specification
    Item_default_specification                  - DEFAULT
    Item_ignore_specification                   - IGNORE

2. Introducing a new virtual method is_evaluable_expression() to
determine if an Item is:
- a normal expression, so its val_xxx()/get_date() methods can be called
- or a just an expression substitute, whose value methods cannot be called.

3. Disallowing Items that are not evalualble expressions in table value
   constructors.
2020-05-19 10:55:39 +04:00
Monty
141cf43e61 Fixed assert in Aria on SHOW PROCEDURE STATUS
MDEV-18457 Assertion `(bitmap->map +
           (bitmap->full_head_size/6*6)) <= full_head_end failed

The problem was that full_head_size was not calculated correctly
in the case when insert_order was inforced, which is the case
for SHOW commands.
2020-05-19 03:42:54 +03:00
Daniel Black
3ea05d0842 largepages: osx compile warning fix
my_is_2pow is only used on linux.

fixes compile warning:

mysys/my_largepage.c:48:23: warning: unused function 'my_is_2pow' [-Wunused-function]

static inline my_bool my_is_2pow(size_t n) { return !((n) & ((n) - 1)); }

                      ^

1 warning generated.
2020-05-18 19:43:50 +04:00
Vladislav Vaintroub
4869e7f4a8 MDEV-22615 system_time_zone may be incorrectly reported as "unknown".
TIME_ZONE_ID_UNKNOWN return code from GetDynamicTimeZoneInformation()
does not mean failure.

It only means, daylight saving dates in the returned strct are not valid.
TIME_ZONE_ID_INVALID means failure, in this case  "unknown" should be returned
2020-05-18 16:39:38 +02:00
Marko Mäkelä
23047d3ed4 Merge 10.4 into 10.5 2020-05-18 17:30:02 +03:00
Vladislav Vaintroub
f86d97c3fe Compiling - improve multithreaded build
In multithreaded build (at least confirmed with Windows ninja and msbuild),
at the end of "sql" target compilation, only 2 processors are used,
compiling either sql_yacc.cc or sql_yacc_ora.cc.
Thus, link of dependent executables or libraries is delayed while build is
underusing the CPU.

Rearrange the source list to improve  parallelism.
2020-05-18 13:00:49 +00:00
Vladislav Vaintroub
8eed9a4648 MDEV-22613 Windows - reduce package size
Liposuction of about 20% package size is achieved by using
linker /OPT:REF,ICF and compile option /Gw
2020-05-18 13:00:49 +00:00
Vladislav Vaintroub
21321fcbbd Windows : small cmake cleanup
Use STRING(APPEND VAR...) rather than SET(VAR "${VAR} ..."), it is shorter
2020-05-18 13:00:49 +00:00
Vladislav Vaintroub
1f952df44a MDEV-22612 Fix -DWITH_ASAN=1 on Windows. 2020-05-18 13:00:49 +00:00
Vladislav Vaintroub
9f8e555895 Fix clang-cl warning 2020-05-18 13:00:49 +00:00
Andrei Elkin
44c8d84908 MDEV-22520 Assertion gathered_length == thd->lex->comment.length failed in binlog_defragment
The assert was caused by early cleanup of a user variable participant
in BINLOG @var,@var where it plays twice. That was unexpected by the base
code to clear its value prematurely.

Fixed with relocating the user var destruction after operations with
its value is over.
2020-05-18 15:14:16 +03:00
Marko Mäkelä
faf6d0ef3f Merge 10.3 into 10.4 2020-05-18 15:05:52 +03:00
Marko Mäkelä
386f168ab3 MDEV-22456 after-merge fix: introduce Atomic_relaxed
In the merge 9e6e43551f
we made Atomic_counter a more generic wrapper of std::atomic
so that dict_index_t would support the implicit assignment operator.

It is better to revert the changes to Atomic_counter and
instead introduce Atomic_relaxed as a generic wrapper to std::atomic.

Unlike Atomic_counter, we will not define operator++, operator+=
or similar, because we want to make the operations more explicit
in the users of Atomic_wrapper, because unlike loads and stores,
atomic read-modify-write operations always incur some overhead.
2020-05-18 15:02:55 +03:00
Jan Lindström
fde94b4cd6 MDEV-21483 : Galera MTR tests failed: galera.MW-328A galera.MW-328B
Enable tests with additional galera output to find out actual
reason for test failures.
2020-05-18 14:21:12 +03:00
Marko Mäkelä
5e12aca57f Merge 10.2 into 10.3 2020-05-18 14:10:11 +03:00
Marko Mäkelä
f0be95b5b0 Merge 10.2 into 10.3 2020-05-18 14:09:34 +03:00
Marko Mäkelä
5b6bcb59ac MDEV-22611 Assertion btr_search_enabled failed during buffer pool resizing
In commit ad6171b91c (MDEV-22456)
we removed the acquisition of the adaptive hash index latch
from the caller of btr_search_update_hash_ref().
The tests innodb.innodb_buffer_pool_resize_with_chunks
and innodb.innodb_buffer_pool_resize
would occasionally fail starting with 10.3,
due to MDEV-12288 causing more purge activity during the test.

btr_search_update_hash_ref(): After acquiring the adaptive hash index
latch, check that the adaptive hash index is still enabled on the page.
2020-05-18 14:04:31 +03:00
Julius Goryavsky
e0ddb077d9 MDEV-22554: galera_sst_mariabackup fails with "Failed to start mysqld.2"
The problem is caused by the operation of netcat streamer and does
not appear on systems where socat is installed. We need to add the
"-N" option for netcat to call shutdown() on the socket when receiving
EOF from STDIN.
2020-05-18 12:17:57 +02:00
Julius Goryavsky
f9d8571f38 MDEV-22554: galera_sst_mariabackup fails with "Failed to start mysqld.2"
The problem is caused by the operation of netcat streamer and does
not appear on systems where socat is installed. We need to add the
"-N" option for netcat to call shutdown() on the socket when receiving
EOF from STDIN.
2020-05-18 12:16:14 +02:00
Julius Goryavsky
ea912d1605 MDEV-22554: galera_sst_mariabackup fails with "Failed to start mysqld.2"
The problem is caused by the operation of netcat streamer and does
not appear on systems where socat is installed. We need to add the
"-N" option for netcat to call shutdown() on the socket when receiving
EOF from STDIN.
2020-05-18 12:14:50 +02:00
Julius Goryavsky
9f294644c4 MDEV-22554: galera_sst_mariabackup fails with "Failed to start mysqld.2"
The problem is caused by the operation of netcat streamer and does
not appear on systems where socat is installed. We need to add the
"-N" option for netcat to call shutdown() on the socket when receiving
EOF from STDIN.
2020-05-18 12:11:59 +02:00
Marko Mäkelä
03aaa7f7a1 Merge 10.2 into 10.3 2020-05-18 10:47:11 +03:00
Marko Mäkelä
d0e3b0eea6 Merge 10.1 into 10.2 2020-05-18 09:43:58 +03:00
Daniel Black
7baa40dffa
MDEV-21976: mtr main.udf - broaden localhost (#1543)
Localhost, depending on the platform can return any
127.0.0.1/8 address.
2020-05-18 09:37:51 +03:00
Otto Kekäläinen
c995090a53 Travis-CI: Remove builds that always fail to make CI useful again
Also clean away dead code that is not used and will never have any use
on the 10.2 branch.
2020-05-17 15:52:35 +03:00
Otto Kekäläinen
8d056affd8 Travis-CI: Shorten deb build log to keep it under 4 MB
There is a 4 MB hard limit on Travis-CI and build output needs to be less
than that. Silencing the 'make install' step gets rid of a lot of
"Installing.." and "Missing.." and removing all mysql-test files will
make the dh_missing warnings much shorter.

Cherry-picked from 41952c85f1.
2020-05-17 15:46:34 +03:00
Otto Kekäläinen
9ddeccc299 Travis-CI: Add missing build dependency dh-exec
Backported from 30b44aaec7.
2020-05-17 15:46:34 +03:00
Varun Gupta
0a5668f512 MDEV-22556: Incorrect result for window function when using encrypt-tmp-files=ON
The issue here is that end_of_file for encrypted temporary IO_CACHE (used by filesort) is updated
using lseek.
Encryption adds storage overhead and hides it from the caller by recalculating offsets and lengths.
Two different IO_CACHE cannot possibly modify the same file
because the encryption key is randomly generated and stored in the IO_CACHE.
So when the tempfiles are encrypted DO NOT use lseek to change end_of_file.

Further observations about updating end_of_file using lseek
1) The end_of_file update is only used for binlog index files
2) The whole point is to update file length when the file was modified via a different file descriptor.
3) The temporary IO_CACHE files can never be modified via a different file descriptor.
4) For encrypted temporary IO_CACHE, end_of_file should not be updated with lseek
2020-05-17 15:26:18 +05:30
Thirunarayanan Balathandayuthapani
4f26aea51b MDEV-21269 Parallel merging of fts index rebuild fails
Problem:
=======
  - During alter rebuild, document read from old table is tokenzied
parallelly by innodb_ft_sort_pll_degree threads and stores it
in respective merge files. While doing the parallel merge, InnoDB
wrongly skips the root level selection of merging buffer records.
So it leads to insertion of merge records in non-ascending order.

Solution:
==========
  Build selection tree for the root level also. So that root of
selection tree can always contain sorted buffer.
2020-05-17 11:43:25 +05:30
Otto Kekäläinen
5d85bc08c6 MDEV-21976: Re-disable main.udf, reverts mistake in merge b30a013142
This test was disabled in a135f0ab88, but
accidentally reverted during the merge in
0186b0a077...b30a013142

Because it was a 3-way merge, the removal is not directly visible.
It can be verified manually though:

  Line not changed:
  git diff a135f0ab88d6..0186b0a07760 mysql-test/unstable-tests

  Line removed:
  git diff a135f0ab88d6..b30a01314217 mysql-test/unstable-tests


This commit fixes the Travis-CI build and makes the main 10.5 git branch
green again on Travis-CI (and whatever else build environment that also
is failing due to this test).

Related: MDEV-22592
2020-05-16 11:48:26 +03:00
Alexander Barkov
54c169a986 Merge remote-tracking branch 'origin/10.2' into 10.3 2020-05-16 12:28:03 +04:00
Otto Kekäläinen
91446c822f Deb: Create empty migrated-from-my.cnf-settings.conf for Buildbot tests
The buildbot.askmonty.org has explicit tests that check that this file
exists, thus to get tests pass, an empty placeholder must be created.
Remove this once CI has been updated not to expect this file.
2020-05-16 10:42:35 +03:00
Otto Kekäläinen
dab5698b3c Deb: Stop using mariadb-service-convert
We can't expect that users want to always convert their mysqld_safe
settings on an upgrade. In its current form it will always run, and that
seems unnecessary on every single installation.

Also the script is buggy, leaks mysqld_safe output into the written file
and since it gets syntax errors the whole mariadb.service will fail to
work.
2020-05-16 10:42:35 +03:00
Otto Kekäläinen
69964c4425 Deb: Add manually dh_systemd_start snippets
These do now show up automatically due to init and systemd customizations,
so the handiest fix is to add them manually. This has been the praxis
in downstream MariaDB packaging for a couple years now, and works fine.

- Ensure service is loaded and started after installation,
  (fixes service start issues in Debian/Ubuntu upgrades where
  otherwise service mysql status stayed stopped)

- Ensure service stopped before removal/purge
  (fixes unstopped processes detected by piuparts)

- Ensure systemd daemon is reloaded after removal/purge when service
  has been removed
2020-05-16 10:42:35 +03:00
Otto Kekäläinen
764dd39ca7 Deb: Add support for legacy init systems again
Partially reverts commit a4cc6fb91f.

While all current versions of Linux have systemd, support for traditional
init.d is still needed e.g. on Linux subsystem on Windows, kFreeBSD and
special variants of Debian/Ubuntu that for other reasons don't have
systemd.

Thus, re-introduce the init file that was remove, but this time with
then name 'mariadb'.

Supporting traditional sysv init in paraller with systemd is easy, since
Debian has facilities for it.

Also simplify and update salsa-ci.yml install/upgrade testing works
for all previous MariaDB and MySQL releases without any excess quirks.

Note that in fresh installs the salsa-ci.yml needs to run command
'service mariadb status' to control the service, while on upgrades
it is enough to run 'service mysql status', since the init.d/mysql
file is left behind from previous install, along with some other
config files such as /etc/default/mysql and /etc/mysql/* stuff.
2020-05-16 10:42:35 +03:00
Alexander Barkov
bf8ae81269 Merge remote-tracking branch 'origin/10.1' into 10.2
Also, adding 10.2 related changes for MDEV-22579
2020-05-16 10:52:08 +04:00
Marko Mäkelä
66f1e288a1 Merge 10.3 into 10.4 2020-05-16 07:54:09 +03:00
Marko Mäkelä
9e6e43551f Merge 10.3 into 10.4
We will expose some more std::atomic internals in Atomic_counter,
so that dict_index_t::lock will support the default assignment operator.
2020-05-16 07:39:15 +03:00
Marko Mäkelä
38d62189a8 Merge 10.2 into 10.3 2020-05-16 06:37:24 +03:00
Marko Mäkelä
4f29d776c7 Merge 10.3 into 10.4 2020-05-16 06:27:55 +03:00
Marko Mäkelä
85651269b6 MDEV-18100: Clean up test 2020-05-16 06:24:09 +03:00
Marko Mäkelä
3f12a5968a MDEV-13626: Make test more robust
In commit b1742a5c95 we forgot
FLUSH TABLES, potentially causing errors for MyISAM system tables.
2020-05-15 22:54:05 +03:00
Marko Mäkelä
fc0960aa96 Merge 10.1 into 10.2 2020-05-15 22:43:33 +03:00
Marko Mäkelä
c8dd411781 MDEV-22544 Inconsistent and Incorrect rw-lock stats
The rw_lock_stats were incorrectly updated.
While global statistics have limited usefulness, we cannot
remove them from a GA version. This contribution is slightly
improving performance in write workloads.
2020-05-15 22:18:11 +03:00
Alexander Barkov
3df297271a MDEV-22579 No error when inserting DEFAULT(non_virtual_column) into a virtual column
The code erroneously allowed both:
INSERT INTO t1 (vcol) VALUES (DEFAULT);
INSERT INTO t1 (vcol) VALUES (DEFAULT(non_virtual_column));

The former is OK, but the latter is not.
Adding a new virtual method in Item:

virtual bool vcol_assignment_allowed_value() const { return false; }

Item_null, Item_param and Item_default_value override it.

Item_default_value overrides it in the way to:
- allow DEFAULT
- disallow DEFAULT(col)
2020-05-15 20:21:54 +04:00
Marko Mäkelä
3d0bb2b7f1 Merge 10.2 into 10.3 2020-05-15 19:11:57 +03:00
Marko Mäkelä
6a6bcc53b8 Merge 10.2 into 10.3 2020-05-15 17:55:01 +03:00
Vladislav Vaintroub
69077dea25 MDEV-22578 thread_pool_info crashes with clang6, using SSE instructions on unaligned memory
Apparently, in stats_reset_table(), the innocuous

  memset(&group->counters, 0, sizeof(group->counters));

is converted by clang to SSE2 instructions.

The problem is that "group" is not correctly aligned,
despite  MY_ALIGNED(CPU_LEVEL1_DCACHE_LINESIZE) in the thread_group_t
declaration.

It is not aligned because it was allocated with my_malloc, since
commit fd9f1638, MDEV-5205. Previously all_groups was a
statically allocated array.

Fix is to remove MY_ALIGNED, and pad the struct instead.
2020-05-15 16:29:06 +02:00
Marko Mäkelä
ad6171b91c MDEV-22456 Dropping the adaptive hash index may cause DDL to lock up InnoDB
If the InnoDB buffer pool contains many pages for a table or index
that is being dropped or rebuilt, and if many of such pages are
pointed to by the adaptive hash index, dropping the adaptive hash index
may consume a lot of time.

The time-consuming operation of dropping the adaptive hash index entries
is being executed while the InnoDB data dictionary cache dict_sys is
exclusively locked.

It is not actually necessary to drop all adaptive hash index entries
at the time a table or index is being dropped or rebuilt. We can let
the LRU replacement policy of the buffer pool take care of this gradually.
For this to work, we must detach the dict_table_t and dict_index_t
objects from the main dict_sys cache, and once the last
adaptive hash index entry for the detached table is removed
(when the garbage page is evicted from the buffer pool) we can free
the dict_table_t and dict_index_t object.

Related to this, in MDEV-16283, we made ALTER TABLE...DISCARD TABLESPACE
skip both the buffer pool eviction and the drop of the adaptive hash index.
We shifted the burden to ALTER TABLE...IMPORT TABLESPACE or DROP TABLE.
We can remove the eviction from DROP TABLE. We must retain the eviction
in the ALTER TABLE...IMPORT TABLESPACE code path, so that in case the
discarded table is being re-imported with the same tablespace identifier,
the fresh data from the imported tablespace will replace any stale pages
in the buffer pool.

rpl.rpl_failed_drop_tbl_binlog: Remove the test. DROP TABLE can
no longer be interrupted inside InnoDB.

fseg_free_page(), fseg_free_step(), fseg_free_step_not_header(),
fseg_free_page_low(), fseg_free_extent(): Remove the parameter
that specifies whether the adaptive hash index should be dropped.

btr_search_lazy_free(): Lazily free an index when the last
reference to it is dropped from the adaptive hash index.

buf_pool_clear_hash_index(): Declare static, and move to the
same compilation unit with the bulk of the adaptive hash index
code.

dict_index_t::clone(), dict_index_t::clone_if_needed():
Clone an index that is being rebuilt while adaptive hash index
entries exist. The original index will be inserted into
dict_table_t::freed_indexes and dict_index_t::set_freed()
will be called.

dict_index_t::set_freed(), dict_index_t::freed(): Note that
or check whether the index has been freed. We will use the
impossible page number 1 to denote this condition.

dict_index_t::n_ahi_pages(): Replaces btr_search_info_get_ref_count().

dict_index_t::detach_columns(): Move the assignment n_fields=0
to ha_innobase_inplace_ctx::clear_added_indexes().
We must have access to the columns when freeing the
adaptive hash index. Note: dict_table_t::v_cols[] will remain
valid. If virtual columns are dropped or added, the table
definition will be reloaded in ha_innobase::commit_inplace_alter_table().

buf_page_mtr_lock(): Drop a stale adaptive hash index if needed.

We will also reduce the number of btr_get_search_latch() calls
and enclose some more code inside #ifdef BTR_CUR_HASH_ADAPT
in order to benefit cmake -DWITH_INNODB_AHI=OFF.
2020-05-15 17:23:08 +03:00
Marko Mäkelä
ff66d65a09 Amend af784385b4: Avoid vtable overhead
When neither MSAN nor Valgrind are enabled, declare
Field::mark_unused_memory_as_defined() as an empty inline function,
instead of declaring it as a virtual function.
2020-05-15 17:23:08 +03:00