Commit graph

190554 commits

Author SHA1 Message Date
Marko Mäkelä
d5d8756de3 Merge 10.4 into 10.5 2020-08-20 12:52:44 +03:00
Marko Mäkelä
2fa9f8c53a Merge 10.3 into 10.4 2020-08-20 11:01:47 +03:00
Marko Mäkelä
de0e7cd72a Merge 10.2 into 10.3 2020-08-20 09:12:16 +03:00
Thirunarayanan Balathandayuthapani
a79c257894 MDEV-23452 Assertion `buf_page_get_io_fix(bpage) == BUF_IO_NONE' failed
in buf_page_set_sticky

- Adding os_thread_yield() in buf_page_create() to avoid the continuous
buffer pool mutex acquistions.
2020-08-20 11:38:10 +05:30
Thirunarayanan Balathandayuthapani
e9d6f1c7ac MDEV-23452 Assertion `buf_page_get_io_fix(bpage) == BUF_IO_NONE' failed
in buf_page_set_sticky

commit a1f899a8ab (MDEV-23233) added the
code to make page sticky. So that InnoDB can't allow the page to
be grabbed by other thread while doing lazy drop of ahi.

But the block could be in flush list and it could have io_fix value
as BUF_IO_WRITE. It could lead to the failure in buf_page_set_sticky().

buf_page_create(): If btr_search_drop_page_hash_index() must be invoked,
take x-latch on the block. If the block io_fix value is other than
BUF_IO_NONE, release the buffer pool mutex and page hash lock and
wait for I/O to complete.
2020-08-20 11:34:53 +05:30
Marko Mäkelä
22c4a7512f MDEV-23514 Race conditions between ROLLBACK and ALTER TABLE
Since commit 1509363970 (MDEV-23484)
the rollback of InnoDB transactions is no longer protected by
dict_operation_lock. Removing that protection revealed a race
condition between transaction rollback and the rollback of an
online table-rebuilding operation (OPTIMIZE TABLE, or any online
ALTER TABLE that is rebuilding the table).

row_undo_mod_clust(): Re-check dict_index_is_online_ddl() after
acquiring index->lock, similar to how row_undo_ins_remove_clust_rec()
is doing it. Because innobase_online_rebuild_log_free() is holding
exclusive index->lock while invoking row_log_free(), this re-check
will ensure that row_log_table_low() will not be invoked when
index->online_log=NULL.

A different race condition is possible between the rollback of a
recovered transaction and the start of online secondary index creation.
Because prepare_inplace_alter_table_dict() is not acquiring an InnoDB
table lock in this case, and because recovered transactions are not
covered by metadata locks (MDL), the dict_table_t::indexes could be
modified by prepare_inplace_alter_table_dict() while the rollback of
a recovered transaction is being executed. Normal transactions would
be covered by MDL, and during prepare_inplace_alter_table_dict() we
do hold MDL_EXCLUSIVE, that is, an online ALTER TABLE operation may
not execute concurrently with other transactions that have accessed
the table.

row_undo(): To prevent a race condition with
prepare_inplace_alter_table_dict(), acquire dict_operation_lock
for all recovered transactions. Before MDEV-23484 we used to acquire
it for all transactions, not only recovered ones.

Note: row_merge_drop_indexes() would not invoke
dict_index_remove_from_cache() while transactional locks
exist on the table, or while any thread is holding an open table handle.
OK, it does that for FULLTEXT INDEX, but ADD FULLTEXT INDEX is not
supported as an online operation, and therefore
prepare_inplace_alter_table_dict() would acquire a table S lock,
which cannot succeed as long as recovered transactions on the table
exist, because they would hold a conflicting IX lock on the table.
2020-08-20 08:34:55 +03:00
Marko Mäkelä
bfba2bce6a Merge 10.1 into 10.2 2020-08-20 06:00:36 +03:00
Marko Mäkelä
b205e478a3 Work around MDEV-23416 (change Warning to Note)
Ever since MDEV-15053 changed something in the page flushing,
we are occasionally observing pending I/O for a data file that
is about to be deleted.

fil_check_pending_io(): Change the Warning to a note.

This message was already made less frequent in
commit dcc0baf540 (10.5.4)
and commit 65f831d17c (10.3.24, 10.4.14).
2020-08-19 15:47:11 +03:00
Daniele Sciascia
f8bf5b0f84 MDEV-23466 SIGABRT on SELECT WSREP_LAST_SEEN_GTID
SELECT WSREP_LAST_SEEN_GTID aborts the server if no provider is
loaded.
2020-08-19 13:12:00 +03:00
Daniele Sciascia
fe3284b2cc MDEV-23092 SIGABRT when setting invalid wsrep_provider
Some invalid wsrep_provider paths may be interpreted as a valid
directory. For example '/invalid/libgalera_smm.so' with UTF character
set is interpreted as '/', which is a valid directory. A early check
that wsrep_provider should not be a directory fixes it.
2020-08-19 13:12:00 +03:00
Daniele Sciascia
09dd06f14a MDEV-22443 wsrep::runtime_error on START TRANSACTION
This happens with global wsrep_on disabled and local wsrep_on enabled.
The fix consists in avoiding sync wait when global wsrep_on is
disabled.
2020-08-19 13:12:00 +03:00
Marko Mäkelä
309302a3da MDEV-23475 InnoDB performance regression for write-heavy workloads
In commit fe39d02f51 (MDEV-20638)
we removed some wake-up signaling of the master thread that should
have been there, to ensure a steady log checkpointing workload.

Common sense suggests that the commit omitted some necessary calls
to srv_inc_activity_count(). But, an attempt to add the call to
trx_flush_log_if_needed_low() as well as to reinstate the function
innobase_active_small() did not restore the performance for the
case where sync_binlog=1 is set.

Therefore, we will revert the entire commit in MariaDB Server 10.2.
In MariaDB Server 10.5, adding a srv_inc_activity_count() call to
trx_flush_log_if_needed_low() did restore the performance, so we
will not revert MDEV-20638 across all versions.
2020-08-19 11:18:56 +03:00
Daniele Sciascia
8a6a084578 Re-record MTR tests galera_3nodes.galera_join_with_cc_{A|B|C} 2020-08-19 10:34:25 +03:00
Marko Mäkelä
1509363970 MDEV-23484 Rollback unnecessarily acquires dict_operation_lock for every row
InnoDB transaction rollback includes an unnecessary work-around for
a data corruption bug that was fixed by me in MySQL 5.6.12
mysql/mysql-server@935ba09d52
and ported to MariaDB 10.0.8 by
commit c291ddfdf7
in 2013 and 2014, respectively.

By acquiring and releasing dict_operation_lock in shared mode,
row_undo() hopes to prevent the table from being dropped while
the undo log record is being rolled back. But, thanks to mentioned fix,
debug assertions (that we are adding) show that the rollback is
protected by transactional locks (table IX lock, in addition to
implicit or explicit exclusive locks on the records that had been modified).

Because row_drop_table_for_mysql() would invoke
row_add_table_to_background_drop_list() if any locks exist on the table,
the mere existence of locks (which is guaranteed during ROLLBACK) is
enough to protect the table from disappearing. Hence, acquiring and
releasing dict_operation_lock for every row that is being rolled back is
unnecessary.

row_undo(): Remove the unnecessary acquisition and release of
dict_operation_lock.

Note: row_add_table_to_background_drop_list() is mostly working around
bugs outside InnoDB:
MDEV-21175 (insufficient MDL protection of FOREIGN KEY operations)
MDEV-21602 (incorrect error handling of CREATE TABLE...SELECT).
2020-08-18 17:30:34 +03:00
Marko Mäkelä
4c50120d14 MDEV-23474 InnoDB fails to restart after SET GLOBAL innodb_log_checksums=OFF
Regretfully, the parameter innodb_log_checksums was introduced
in MySQL 5.7.9 (the first GA release of that series) by
mysql/mysql-server@af0acedd88
which partly replaced a parameter that had been introduced in 5.7.8
mysql/mysql-server@22ba38218e
as innodb_log_checksum_algorithm.

Given that the CRC-32C operations are accelerated on many processor
implementations (AMD64 with SSE4.2; since MDEV-22669 also on IA-32
with SSE4.2, POWER 8 and later, ARMv8 with some extensions)
and by lookup tables when only generic SISD instructions are available,
there should be no valid reason to disable checksums.

In MariaDB 10.5.2, as a preparation for MDEV-12353, MDEV-19543 deprecated
and ignored the parameter innodb_log_checksums altogether. This should
imply that after a clean shutdown with innodb_log_checksums=OFF one
cannot upgrade to MariaDB Server 10.5 at all.

Due to these problems, let us deprecate the parameter innodb_log_checksums
and honor it only during server startup.
The command SET GLOBAL innodb_log_checksums will always set the
parameter to ON.
2020-08-18 16:46:07 +03:00
Marko Mäkelä
064bfbaf06 MDEV-23499 Assertion c.same_type(*o) failed
dict_col_t::same_encoding(), dict_col_t::same_format(): Allow
an instantaneous change of a column to a compatible encoding,
just like ha_innobase::can_convert_string() and similar functions do.
2020-08-18 15:14:12 +03:00
Thirunarayanan Balathandayuthapani
8268f26605 MDEV-22934 Table disappear after two alter table command
Problem:
=======
InnoDB drops the column which has foreign key relations on it. So it
tries to load the foreign key during rename process of copy algorithm
even though the foreign_key_check is disabled.

Solution:
========
During alter copy algorithm, InnoDB ignores the error while loading
the foreign key constraint if foreign key check is disabled. It
should throw the warning about failure of the foreign key constraint
when foreign key check is disabled.
2020-08-18 15:05:23 +05:30
Thirunarayanan Balathandayuthapani
362b18c536 MDEV-23380 InnoDB reads a page from disk despite parsing MLOG_INIT_FILE_PAGE2 record
This problem is caused by 6697135c6d
(MDEV-21572). During recovery, InnoDB prefetches the siblings of
change buffer index leaf page. It does asynchronous page read
and recovery scenario wasn't handled in buf_read_page_background().
It leads to the refusal of startup of the server.

Solution:
=========
  InnoDB shouldn't allow the change buffer index page siblings
to be prefetched.
2020-08-18 14:59:16 +05:30
Oleksandr Byelkin
ece0b0623c MDEV-23491: __bss_start breaks compilation of various platforms
Remove __bss_start & Co, because systen call "write" check buffer address and return EFAULT if it is wrong.
2020-08-18 08:56:35 +02:00
Julius Goryavsky
5796021174 MDEV-21039: Server fails to start with unknown mysqld_safe options
Adding any unknown option to the "[mysqld_safe]" section makes
mysqld impossible to start with mysqld_multi. For example, after
adding the unknown option "numa_interleave" to the "[mysqld_safe]"
section, mysqld_multi exits with the following diagnostics:

[ERROR] /usr/local/mysql/bin/mysqld: unknown option '--numa_interleave'

To get rid of this behavior, this patch by default adds the "--loose-"
prefix to all unknown (for mysqld_safe) options. This behavior can be
enabled explicitly with the --ignore-unknown option and disabled with
the --no-ignore-unknown option.
2020-08-18 03:13:46 +02:00
Kentoku SHIBA
5c8a1249dd MDEV-20827 Wrong param parsing in spider_direct_sql() when param contain comma 2020-08-17 21:04:02 +09:00
Kentoku SHIBA
314a90e12b MDEV-20827 Wrong param parsing in spider_direct_sql() when param contain comma 2020-08-17 21:02:09 +09:00
Eugene Kosov
90c8d773ed MDEV-21251 CHECK TABLE fails to check info_bits of records
btr_validate_index(): do not stop checking after some level failed.
That way it'll become possible to see errors in leaf pages even when
uppers layers are corrupted too.

page_validate(): check info_bits and status_bits more
2020-08-15 23:05:09 +03:00
Vladislav Vaintroub
010fd61a5f MDEV-23489 Windows zip files 10.4.14 have an embryonic data folder
Fix : Do not INSTALL anything with cmake from data folder.
2020-08-15 17:57:27 +02:00
Daniel Black
b970363acf MDEV-23440: mysql_tzinfo_to_sql to use transactions
Since MDEV-18778, timezone tables get changed to innodb
to allow them to be replicated to other galera nodes.

Even without galera, timezone tables could be declared innodb.
With the standalone innodb tables, the mysql_tzinfo_to_sql takes
approximately 27 seconds.

With the transactions enabled in this patch, 1.2 seconds is
the approximate load time.

While explicit checks for the engine of the time zone tables could be
done, or checks against !opt_skip_write_binlog, non-transactional
storage engines will just ignore the transactional state without
even a warning so its safe to enact globally.

Leap seconds are pretty much ignored as they are a single insert
statement and have gone out of favour as they have caused MariaDB
stalls in the past.
2020-08-15 14:02:05 +10:00
Kentoku SHIBA
303212d4b6 MDEV-19794 Spider crash with XA 2020-08-14 01:54:24 +09:00
Jan Lindström
7df4706619 Fix Windows compiler error. 2020-08-14 13:34:19 +03:00
Jan Lindström
a7a9f44f8c MDEV-22543 : Galera SST donation fails, FLUSH TABLES WITH READ LOCK times out
During SST we need to let FTWRL to use normal timeout method
even when client is disconnected.
2020-08-14 08:32:36 +03:00
Kentoku SHIBA
ad1a3ce418 MDEV-19794 Spider crash with XA 2020-08-14 01:47:48 +09:00
Alexey Botchkov
b01c426146 MDEV-19275 Provide SQL service to plugins.
Protocol_local fixed so it can be used now.
Some Protocol:: methods made virtual so they can adapt.
as well as net_ok and net_send_error functions.
execute_sql_string function is exported to the plugins.
To be changed with the mysql_use_result.
2020-08-14 21:04:25 +04:00
Sergei Petrunia
68cba09173 Add QT_ITEM_IDENT_DISABLE_DB_TABLE_NAMES flag for Item::print
It allows to skip db and table name when printing table column references.
2020-08-14 14:45:36 +03:00
Sergei Petrunia
16d8d18907 Basic Optimizer Trace support for table condition pushdown
Print the condition being pushed.
2020-08-14 14:43:26 +03:00
Jan Lindström
eab219d594 MDEV-22543 : Galera SST donation fails, FLUSH TABLES WITH READ LOCK times out
During SST we need to let FTWRL to use normal timeout method
even when client is disconnected.
2020-08-14 13:35:42 +03:00
Marko Mäkelä
cf87f3e08c Merge 10.4 into 10.5 2020-08-14 11:33:35 +03:00
Marko Mäkelä
1bf77cde5c MDEV-23162: Fix clang -Winconsistent-missing-override 2020-08-14 11:33:06 +03:00
Alexander Barkov
48cbb2c021 MDEV-23478 Improve Protocol performance for numeric data - version 2
The previous commit was erroneously marked as MDEV-23162.
The correct issue is MDEV-23478.
2020-08-14 11:04:39 +04:00
Alexander Barkov
c55f24cd99 MDEV-23162 Improve Protocol performance for numeric data
An alternative implementation (replacing the one based on repertoire).

This implementation makes Field send itself to Protocol_text using
data type specific Protocol methods rather than field->val_str()
followed by protocol_text->store_str().

As now Field sends itself in the same way to all protocol types
(e.g. Protocol_binary, Protocol_text, Protocol_local),
the method Field::send_binary() was renamed just to Field::send().

Note, this change introduces symmetry between Field and Item,
because Items also send themself using a single method Item::send(),
which is used for *all* protocol types.

Performance improvement is achieved by the fact that Protocol_text
implements these data type specific methods using store_numeric_string_aux()
rather than store_string_aux(). The conversion now happens only when
character_set_results is not ASCII compatible character sets
(e.g. UCS2, UTF16, UTF32).

In the old code (before any MDEV-23162 work, e.g. as of 10.5.4),
Protocol_text::store(Field*) used val_str() for all data types.
So the execution went through the character set conversion routines
even for numeric and temporal data types.

Benchmarking summary (see  details in MDEV-23478):

The new approach stably demonstrates additional improvement comparing
to the previous implementation (the smaller time - the better):

Original   - the commit before MDEV-23162
             be98036f25

        1m9.336s
        1m9.290s
        1m9.300s

MDEV-23162 - the repertoire optimization

        1m6.101s
        1m5.988s
        1m6.264s

MDEV-23478 - this commit

        1m2.150s
        1m2.079s
        1m2.099s
2020-08-14 10:07:03 +04:00
Alexander Barkov
f1a9700fec Revert "MDEV-23162 Improve Protocol performance for numeric data"
This reverts commit eb2eaba7fd.

A different implementation of MDEV-23162 is coming.
2020-08-14 09:14:07 +04:00
Alexander Barkov
e96f66b93d MDEV-23270 Remove a String parameter from Protocol::store(double/float) 2020-08-14 09:14:07 +04:00
Otto Kekäläinen
0ac8e2cfdb Add a Github CODEOWNERS file to project
As the first entry, define user 'ottok' (Otto Kekäläinen) as the person
who by default gets assigned all PRs that modify the debian/ contents.
2020-08-14 09:03:34 +10:00
Marko Mäkelä
2f7b37b021 Merge 10.3 into 10.4, except MDEV-22543
Also, fix GCC -Og -Wmaybe-uninitialized in run_backup_stage()
2020-08-13 18:48:41 +03:00
Marko Mäkelä
b811c6ecc7 Fix GCC 10.2.0 -Og -Wmaybe-uninitialized
Fix some more cases after merging
commit 31aef3ae99.
Some warnings look possibly genuine, others are clearly bogus.
2020-08-13 18:21:30 +03:00
Marko Mäkelä
4bd56a697f Merge 10.2 into 10.3 2020-08-13 18:18:25 +03:00
Marko Mäkelä
3e617b8bef Merge 10.1 into 10.2 2020-08-13 17:50:40 +03:00
Marko Mäkelä
7c2aad6be2 MDEV-23463 fil_page_decompress() debug check wastes 128KiB of stack
fil_page_decompress(): Remove a rather useless debug check.
We should have test coverage for reading page_compressed pages
from files, either due to buffer pool page eviction or due to
server restarts.

A similar check was removed from fil_space_encrypt() in
commit 0b36c27e0c (MDEV-20307).
2020-08-13 17:43:37 +03:00
Marko Mäkelä
182e2d4a6c Merge 10.1 into 10.2 2020-08-13 07:38:35 +03:00
Vladislav Vaintroub
5eff7c0226 MDEV-23462 Upgrade wizard not offered during 10.5 MSI installation on Windows
There is no mysql_upgrade_wizard target, only mariadb-upgrade-wizard
2020-08-12 21:38:03 +02:00
Vladislav Vaintroub
7541080214 MDEV-23461 mysql_upgrade_wizard.exe differs from mariadb-upgrade-wizard.exe
The post-build custom command to embed Vista elevation manifest into
mariadb-upgrade-wizard.exe seems to do
something nasty to the executable, perhaps it removes and recreates it.

Thus the previously created hardlink mysql_upgrade_wizard is not marked
to require elevation.

Solved by using MANIFESTUAC linker flag, rather than invoke mt.exe.
This avoids an extra post-build step that modifies mariadb-upgrade-wizard.exe
2020-08-12 21:18:21 +02:00
Marko Mäkelä
101ce10d0d MDEV-20672 Inconsistent usage message for innodb_compression_algorithm
The usage message for the innodb_compression_algorithm system variable
did not list snappy, which was added as an optional compression algorithm
in MariaDB 10.1.3 and might actually work since
commit 90c52e5291 (MDEV-12615)
in MariaDB 10.1.24.

Unfortunately, we will include also unavailable compression algorithms
in the list, because ENUM parameters allow numeric values, and we do
not want innodb_compression_algorithm=3 to change meaning depending on
the way how the source code was compiled.
2020-08-12 18:35:21 +03:00
Marko Mäkelä
efd8af535a MDEV-19526 heap number overflow on innodb_page_size=64k
InnoDB only reserves 13 bits for the heap number in the record header,
limiting the heap number to be at most 8191. But, when using
innodb_page_size=64k and secondary index records of 7 bytes each,
it is possible to exceed the maximum heap number.

btr_cur_optimistic_insert(): Let the operation fail if the
maximum number of records would be exceeded.

page_mem_alloc_heap(): Move to the same compilation unit with the
only caller, and let the operation fail if the maximum heap number
has been allocated already.
2020-08-12 18:21:53 +03:00