Commit graph

195664 commits

Author SHA1 Message Date
Sergei Petrunia
8db9aa496c MDEV-28268: Server crashes in Expression_cache_tracker::fetch_current_stats
(cherry-pick into preview-10.9-MDEV-27021-explain tree)

Expression_cache_tmptable object uses an Expression_cache_tracker object
to report the statistics.

In the common scenario, Expression_cache_tmptable destructor sets
tracker->cache=NULL. The tracker object survives after the expression
cache is deleted and one may call cache_tracker->fetch_current_stats()
for it with no harm.

However a degenerate cache with no parameters does not set
tracker->cache=NULL in Expression_cache_tmptable destructor which
results in an attempt to use freed data in the
cache_tracker->fetch_current_stats() call.

Fixed by setting tracker->cache to NULL and wrapping the assignment into
a function.
2022-04-29 11:00:37 +03:00
Sergei Petrunia
3f68c2169e MDEV-28201: Server crashes upon SHOW ANALYZE/EXPLAIN FORMAT=JSON
- Describe the lifetime of EXPLAIN data structures in
  sql_explain.h:ExplainDataStructureLifetime.

- Make Item_field::set_field() call set_refers_to_temp_table()
  when it refers to a temp. table.
- Introduce QT_DONT_ACCESS_TMP_TABLES flag for Item::print.
  It directs Item_field::print to not try access its the
  temp table.
- Introduce Explain_query::notify_tables_are_closed()
  and call it right before the query closes its tables.
- Make Explain data stuctures' print_explain_json() methods
  accept "no_tmp_tbl" parameter which means pass
  QT_DONT_ACCESS_TMP_TABLES when printing items.
- Make Show_explain_request::call_in_target_thread() not call
  set_current_thd(). This wasn't needed as the code inside
  lex->print_explain() uses output->thd anyway. output->thd
  refers to the SHOW command's THD object.
2022-04-29 10:48:26 +03:00
Oleg Smirnov
02c3babdec MDEV-28124 Server crashes in Explain_aggr_filesort::print_json_members
SHOW EXPLAIN/ANALYZE FORMAT=JSON tries to access items that have already been
freed by a call to free_items() during THD::cleanup_after_query().
The solution is to disallow APC calls including SHOW EXPLAIN/ANALYZE
just before the call to free_items().
2022-04-29 10:48:25 +03:00
Oleg Smirnov
a0475cb9ca MDEV-27021 Add explicit indication of SHOW EXPLAIN/ANALYZE.
1. Add explicit indication that the output is produced by
SHOW EXPLAIN/ANALYZE FORMAT=JSON command.
2. Remove useless "r_total_time_ms" field from SHOW ANALYZE FORMAT=JSON
output when there is no timed statistics gathered.
3. Add "r_query_time_in_progress_ms" to the output of SHOW ANALYZE FORMAT=JSON.
2022-04-29 10:48:25 +03:00
Oleg Smirnov
d1a1ad4c28 MDEV-27021 Add support of FORMAT=JSON for SHOW ANALYZE 2022-04-29 10:48:25 +03:00
Oleg Smirnov
e7fcd496d4 MDEV-27021 Implement SHOW ANALYZE command 2022-04-29 10:48:20 +03:00
Oleg Smirnov
328684833b MDEV-10000 Add EXPLAIN [FORMAT=JSON] FOR CONNECTION syntax support
EXPLAIN FOR CONNECTION is a MySQL-compatible syntax for SHOW EXPLAIN.
This commit also adds support for FORMAT=JSON to SHOW EXPLAIN,
so the possible options to get JSON-formatted output are:
- SHOW EXPLAIN FORMAT=JSON FOR $con
- EXPLAIN FORMAT=JSON FOR CONNECTION $con
2022-04-29 10:47:03 +03:00
Brandon Nesterenko
51b28b24ca MDEV-28435: rpl.rpl_mysqlbinlog_slave_consistency fails intermittently on tables comparison
Problem:
========
The test logic checked for the wrong condition to validate that the
slave had caught up with the master. Specifically, it used the
thread stage of the IO and SQL thread to be in the “Waiting for
master to send event” and “Slave has read all relay log; waiting for
more updates” states, respectively. The problem exposed by this MDEV
is that, this state is also the initial slave state before reading
data from the primary (whereas the intended state was having already
read all available events from the primary and now waiting for new
events). This made the MTR test validate data that it had not yet
received, and thereby fail.

Solution:
========
Instead of using the IO/SQL thread states, use the existing helper
functions save_master_gtid.inc and sync_with_master_gtid.inc. Note
that the test result file also needed to be updated to reflect
this fix.

Special thanks to Angelique Sklavounos for pointing out that
--stop-position was not specified in any buildbot failures, as this
led to an IF block in the MTR test that was the source of the test
failure.

Reviewed By
============
Andrei Elkin <andrei.elkin@mariadb.com>
2022-04-28 11:46:08 -06:00
Marko Mäkelä
504a3b32f6 Merge 10.8 into 10.9 2022-04-28 15:54:03 +03:00
Marko Mäkelä
133c2129cd Merge 10.7 into 10.8 2022-04-27 10:43:00 +03:00
Marko Mäkelä
638afc4acf Merge 10.6 into 10.7 2022-04-26 18:59:40 +03:00
Marko Mäkelä
2c005261cb MDEV-26753 Assertion state == TRX_STATE_PREPARED ||... failed
dict_stats_save(): Do not attempt to commit an already committed
transaction.
2022-04-26 18:09:14 +03:00
Marko Mäkelä
2ca1123464 MDEV-26217 Failing assertion: list.count > 0 in ut_list_remove or Assertion `lock->trx == this' failed in dberr_t trx_t::drop_table
This follows up the previous fix in
commit c3c53926c4 (MDEV-26554).

ha_innobase::delete_table(): Work around the insufficient
metadata locking (MDL) during DML operations by acquiring exclusive
InnoDB table locks on all child tables. Previously, this was only
done on TRUNCATE and ALTER.

ibuf_delete_rec(), btr_cur_optimistic_delete(): Do not invoke
lock_update_delete() during change buffer operations.
The revised trx_t::commit(std::vector<pfs_os_file_t>&) will
hold exclusive lock_sys.latch while invoking fil_delete_tablespace(),
which in turn may invoke ibuf_delete_rec().

dict_index_t::has_locking(): A new predicate, replacing the dummy
!dict_table_is_locking_disabled(index->table). Used for skipping lock
operations during ibuf_delete_rec().

trx_t::commit(std::vector<pfs_os_file_t>&): Release the locks
and remove the table from the cache while holding exclusive
lock_sys.latch.

trx_t::commit_in_memory(): Skip release_locks() if dict_operation holds.

trx_t::commit(): Reset dict_operation before invoking commit_in_memory()
via commit_persist().

lock_release_on_drop(): Release locks while lock_sys.latch is
exclusively locked.

lock_table(): Add a parameter for a pointer to the table.
We must not dereference the table before a lock_sys.latch has
been acquired. If the pointer to the table does not match the table
at that point, the table is invalid and DB_DEADLOCK will be returned.

row_ins_foreign_check_on_constraint(): Improve the checks.
Remove a bogus DB_LOCK_WAIT_TIMEOUT return that was needed
before commit c5fd9aa562 (MDEV-25919).

row_upd_check_references_constraints(),
wsrep_row_upd_check_foreign_constraints(): Simplify checks.
2022-04-26 18:09:03 +03:00
Vladislav Vaintroub
ec85e7b196 MDEV-28340 Add tests for versioning, sequence, I_S table 2022-04-26 15:56:58 +02:00
Federico Razzoli
489011dd0f MDEV-28340 Don't try to create temptables in system DBs, support table_type='SYSTEM VIEW' 2022-04-26 15:56:58 +02:00
Federico Razzoli
3fe656e629 MDEV-28340 On table_type='SYSTEM VERSIONED' return 'BASE TABLE' instead of failing 2022-04-26 15:56:58 +02:00
Federico Razzoli
81862dbb08 MDEV-28340 sys.table_exists: support table_type=SEQUENCE 2022-04-26 15:56:58 +02:00
Marko Mäkelä
e135edec3a Merge 10.5 into 10.6 2022-04-26 15:21:20 +03:00
Marko Mäkelä
7725f870b0 MDEV-15250 fixup: Remove MY_GNUC_PREREQ
The macro MY_GNUC_PREREQ() was used for testing for some minor
GCC 4 versions before GCC 4.8.5, which is the oldest version
that supports C++11, which we depend on ever since
commit d9613b750c
2022-04-26 15:13:39 +03:00
Thirunarayanan Balathandayuthapani
7c0b9c6020 MDEV-15250 UPSERT during ALTER TABLE results in 'Duplicate entry' error for alter
- InnoDB should avoid bulk insert operation when table has active
DDL. Because bulk insert writes only one undo log as TRX_UNDO_EMPTY
and logging of concurrent DML happens at commit time uses undo log
record to parse and get the value and operation.

- Removed ROW_T_EMPTY, ROW_OP_EMPTY and their associated functions
and also the test case which tries to log the ROW_OP_EMPTY
when table has active DDL.
2022-04-26 16:22:56 +05:30
Rucha Deodhar
43fa8e0b8f MDEV-28319: Assertion `cur_step->type & JSON_PATH_KEY' failed in json_find_path
Analysis: When trying to find path and handling the match for path,
value at current index is not set to 0 for array_counters. This causes wrong
current step value which eventually causes wrong cur_step->type value.
Fix: Set the value at current index for array_counters to 0.
2022-04-26 16:13:19 +05:30
Rucha Deodhar
ee5966c754 MDEV-26695: Number of an invalid row is not calculated for table value
constructor

Analysis: counter does not increment while sending rows for table value
constructor and so row_number assumes the default value (0 in this case).
Fix: Increment the counter to avoid counter using default value.
2022-04-26 14:27:50 +05:30
Rucha Deodhar
4730a6982f MDEV-28350: Test failing on buildbot with UBSAN
Analysis: There were two kinds of failing tests on buildbot with UBSAN.
1) runtime error: signed integer overflow and
2) runtime error: load of value is not valid value for type
Signed integer overflow was occuring because addition of two integers
(size of json array + item number in array) was causing overflow in
json_path_parts_compare. This overflow happens because a->n_item_end
wasn't set.
The second error was occuring because c_path->p.types_used is not
initialized but the value is used later on to check for negative path index.
Fix: For signed integer overflow, use a->n_item_end only in case of range
so that it is set.
2022-04-26 13:59:43 +05:30
Rucha Deodhar
3716eaff4e MDEV-28326: Server crashes in json_path_parts_compare
Analysis: When trying to compare json paths, the array_sizes variable is
NULL when beginning. But trying to access address by adding to the NULL
pointer while recursive calling json_path_parts_compare() for handling
double wildcard, it causes undefined behaviour and the array_sizes
variable eventually becomes non-null (has some address).
This eventually results in crash.
Fix: If array_sizes variable is NULL then pass NULL recursively as well.
2022-04-26 12:37:11 +05:30
Julius Goryavsky
cad792c686 MDEV-28377: galera_as_slave_nonprim bind: Address already in use
This commit fixes a crash reported as MDEV-28377 and a number
of other crashes in automated tests with mtr that are related
to broken .cnf files in galera and galera_3nodes suites, which
happened when automatically migrating MDEV-26171 from 10.3 to
subsequent higher versions.
2022-04-26 06:11:33 +02:00
Tuukka Pasanen
375b8f40ce MDEV-27033: Remove version suffix from Debian packages
Remove version suffix from Debian packages (for example mariadb-server-10.9)
because installing suffixed package removes older version
of package even if it's suffixed (for example mariadb-server-10.7)

This make also Debian package management easier in future MariaDB
version iterations because there is no need for stacking
Conlicts/Breaks/Replaces-parameters in every new major release
2022-04-26 12:46:39 +10:00
Thirunarayanan Balathandayuthapani
4b80c11f52 MDEV-15250 UPSERT during ALTER TABLE results in 'Duplicate entry' error for alter
- InnoDB DDL results in `Duplicate entry' if concurrent DML throws
duplicate key error. The following scenario explains the problem

connection con1:
  ALTER TABLE t1 FORCE;

connection con2:
  INSERT INTO t1(pk, uk) VALUES (2, 2), (3, 2);

In connection con2, InnoDB throws the 'DUPLICATE KEY' error because
of unique index. Alter operation will throw the error when applying
the concurrent DML log.

- Inserting the duplicate key for unique index logs the insert
operation for online ALTER TABLE. When insertion fails,
transaction does rollback and it leads to logging of
delete operation for online ALTER TABLE.
While applying the insert log entries, alter operation
encounters 'DUPLICATE KEY' error.

- To avoid the above fake duplicate scenario, InnoDB should
not write any log for online ALTER TABLE before DML transaction
commit.

- User thread which does DML can apply the online log if
InnoDB ran out of online log and index is marked as completed.
Set online log error if apply phase encountered any error.
It can also clear all other indexes log, marks the newly
added indexes as corrupted.

- Removed the old online code which was a part of DML operations

commit_inplace_alter_table() : Does apply the online log
for the last batch of secondary index log and does frees
the log for the completed index.

trx_t::apply_online_log: Set to true while writing the undo
log if the modified table has active DDL

trx_t::apply_log(): Apply the DML changes to online DDL tables

dict_table_t::is_active_ddl(): Returns true if the table
has an active DDL

dict_index_t::online_log_make_dummy(): Assign dummy value
for clustered index online log to indicate the secondary
indexes are being rebuild.

dict_index_t::online_log_is_dummy(): Check whether the online
log has dummy value

ha_innobase_inplace_ctx::log_failure(): Handle the apply log
failure for online DDL transaction

row_log_mark_other_online_index_abort(): Clear out all other
online index log after encountering the error during
row_log_apply()

row_log_get_error(): Get the error happened during row_log_apply()

row_log_online_op(): Does apply the online log if index is
completed and ran out of memory. Returns false if apply log fails

UndorecApplier: Introduced a class to maintain the undo log
record, latched undo buffer page, parse the undo log record,
maintain the undo record type, info bits and update vector

UndorecApplier::get_old_rec(): Get the correct version of the
clustered index record that was modified by the current undo
log record

UndorecApplier::clear_undo_rec(): Clear the undo log related
information after applying the undo log record

UndorecApplier::log_update(): Handle the update, delete undo
log and apply it on online indexes

UndorecApplier::log_insert(): Handle the insert undo log
and apply it on online indexes

UndorecApplier::is_same(): Check whether the given roll pointer
is generated by the current undo log record information

trx_t::rollback_low(): Set apply_online_log for the transaction
after partially rollbacked transaction has any active DDL

prepare_inplace_alter_table_dict(): After allocating the online
log, InnoDB does create fulltext common tables. Fulltext index
doesn't allow the index to be online. So removed the dead
code of online log removal

Thanks to Marko Mäkelä for providing the initial prototype and
Matthias Leich for testing the issue patiently.
2022-04-25 18:52:19 +05:30
Marko Mäkelä
1a66e3f861 Skip main.sp-no-valgrind WITH_MSAN due to result diff 2022-04-25 14:36:00 +03:00
Marko Mäkelä
cba13079da Remove redundant innodb-page_compression_ tests
These were replaced with innodb.innodb_page_compressed
in commit 35095c458b
2022-04-25 10:42:57 +03:00
Marko Mäkelä
35095c458b Clean up the page_compressed tests
It suffices to test compression with one record. Restarting the
server is not really needed; we are exercising the log based recovery
in other tests, such as mariabackup.page_compression_level.
2022-04-25 10:19:12 +03:00
Marko Mäkelä
4faef6e240 Cleanup: Remove IF_VALGRIND
The purpose of the compress() wrapper my_compress_buffer() was twofold:
silence Valgrind warnings about uninitialized memory access before
zlib 1.2.4, and have PERFORMANCE_SCHEMA instrumentation of some zlib
related memory allocation. Because of PERFORMANCE_SCHEMA, we cannot
trivially replace my_compress_buffer() with compress().

az_open(): Remove a crc32() call. Any CRC of the empty string is 0.
2022-04-25 09:40:40 +03:00
Marko Mäkelä
232af0c7bf Do not disable --symbolic-links on Valgrind (or MSAN)
The option --symbolic-links was originally disabled by default under
Purify (and later Valgrind) in 51156c5af2
without any explanation.
2022-04-25 09:36:30 +03:00
Alexander Barkov
4ed30b2ac5 MDEV-27690 Crash on CHARACTER SET csname COLLATE DEFAULT in column definition
Adding a 10.6 specific test from the MDEV
2022-04-25 09:32:32 +04:00
Marko Mäkelä
c009ce7dd0 MDEV-27094 Debug builds include useless InnoDB "disabled" options
This is a backport of commit 4489a89c71
in order to remove the test innodb.redo_log_during_checkpoint
that would cause trouble in the DBUG subsystem invoked by
safe_mutex_lock() via log_checkpoint(). Before
commit 7cffb5f6e8
these mutexes were of different type.

The following options were introduced in
commit 2e814d4702 (mariadb-10.2.2)
and have little use:

innodb_disable_resize_buffer_pool_debug had no effect even in
MariaDB 10.2.2 or MySQL 5.7.9. It was introduced in
mysql/mysql-server@5c4094cf49
to work around a problem that was fixed in
mysql/mysql-server@2957ae4f99
(but the parameter was not removed).

innodb_page_cleaner_disabled_debug and innodb_master_thread_disabled_debug
are only used by the test innodb.redo_log_during_checkpoint
that will be removed as part of this commit.

innodb_dict_stats_disabled_debug is only used by that test,
and it is redundant because one could simply use
innodb_stats_persistent=OFF or the STATS_PERSISTENT=0 attribute
of the table in the test to achieve the same effect.
2022-04-22 12:48:40 +03:00
Marko Mäkelä
6948abb94c Cleanup: Remove fil_names_write_bogus
Ever since commit 685d958e38
some Perl code in the test mariabackup.huge_lsn is writing names of
non-existing files to the InnoDB redo log and testing the recovery.
We do not need any debug instrumentation to duplicate that test.
2022-04-22 12:22:28 +03:00
Tuukka Pasanen
b884410a0e MDEV-28232: Add missing conflict and breaks for MariaDB 10.7
There is missing conflict and breaks for MariaDB 10.7 which
can cause upgrading from 10.6 not be clean and end up with
errors on Debian.
2022-04-22 17:25:41 +10:00
Tuukka Pasanen
7f28cbf6b7 MDEV-28300: Add upgrade test from MariaDB version 10.7 to 10.8
Salsa-CI does not have upgrade test from 10.7 to 10.8. Add that
to make sure nothing gets broke during update.
2022-04-22 17:00:27 +10:00
Marko Mäkelä
fae0ccad6e Merge 10.5 into 10.6 2022-04-21 17:46:40 +03:00
Marko Mäkelä
620c55e708 Merge 10.4 into 10.5 2022-04-21 15:33:50 +03:00
Vlad Lesin
1b558dd462 MDEV-27919 mariabackup --log-copy-interval is measured in millisecondss in 10.5 and in microseconds in 10.6
Multiply polling interval by 1000.
2022-04-21 15:24:59 +03:00
Marko Mäkelä
fdec842fd7 MDEV-28371 Assertion fold == id.fold() failed in buf_flush_check_neighbor()
Due to 32-bit arithmetics, SRV_TMP_SPACE_ID page number 0x200002 would be
folded to 0, which is incompatible with the assumption that was made in
commit 7cffb5f6e8 (MDEV-23399).

page_id_t::fold(): Compute in the native word width instead of uint32_t.
On 64-bit platforms, an alternative would be to return the 64-bit m_id
directly, but that was measured to cause a performance regression.

fil_space_t::open(): Invoke fil_node_t::find_metadata() when the
tablespace is being created. In this way, we will actually detect
that the temporary tablespace resides on SSD. (During database
creation, also the system tablespace will correctly be detected as
residing on SSD.)
2022-04-21 14:09:23 +03:00
Daniel Black
f0c52bfe3b MDEV-28263: mariadb-tzinfo-to-sql (postfix)
I missed include/no_protocol.inc already existed.
2022-04-21 19:13:58 +10:00
Daniel Black
c47069ead7 MDEV-28313 InnoDB transactions are not aligned at cache lines (postfix)
aarch64 RHEL7 and Centos7 internal compiler error here, so we carry
another pragma enchantment like buf0lru.cc and row0ins.cc.
2022-04-21 19:12:24 +10:00
Marko Mäkelä
aec856073d WolfSSL v5.2.0-stable 2022-04-21 12:02:36 +03:00
Marko Mäkelä
f84b5d782a Fix clang -Wunused-but-set-variable 2022-04-21 11:35:07 +03:00
Marko Mäkelä
394784095e Merge 10.3 into 10.4 2022-04-21 11:33:59 +03:00
Marko Mäkelä
4730314a70 MDEV-28369 ibuf_bitmap_mutex is an unnecessary contention point
The only purpose of ibuf_bitmap_mutex is to prevent a deadlock between
two concurrent invocations of ibuf_update_free_bits_for_two_pages_low()
on the same pair of bitmap pages, but in opposite order.
The mutex is unnecessarily serializing the execution of the function
even when it is being invoked on totally different tablespaces.
To avoid deadlocks, it suffices to ensure that the two page latches
are being acquired in a deterministic (sorted) order.
2022-04-21 09:15:18 +03:00
Daniel Black
580cbd18b3 Merge branch 10.4 into 10.5
A few of constaint -> constraint
2022-04-21 15:47:03 +10:00
Daniel Black
b8398ee1fd MDEV-21208: mysql_tzinfo_to_sql does not work in strict mode
Remove Warning that occured by doing an ALTER TABLE ... ORDER
BY on an InnoDB table.

Reviewed by Brandon Nesterenko
2022-04-21 14:59:29 +10:00
Daniel Black
13e77930e6 MDEV-28263: mariadb-tzinfo-to-sql improve wsrep and binlog cases
The --skip-write-binlog message was confusing that it only had
an effect if the galera was enabled. There are uses beyond galera
so we apply SET SESSION SQL_LOG_BIN=0 as implied by the option
without being conditional on the wsrep status.

We also with --skip-write-binlog actually check the session @@WSREP_ON
variable rather than the global server variable.

Since 10.6, the wsrep_mode could replicate Aria and MyISAM, in which
case no change to innodb and back is needed.

By removing the conditions, we can use LOCK TABLES in a general case
improving the load speed of Aria (MDEV-23326), regardless of the
skip-write-binlog flag. The only case where we don't use LOCK TABLES is
when we are replicating via Innodb, because wsrep_on=1 and wsrep_mode
doesn't contain REPLICATE_ARIA{,MYISAM}. This uses an Innodb transaction
instead. When replicating via InnoDB we change the table engine type
back to what it was originally.

By removing the \d and other syntax that requires parsing by
the mariadb client, we can use the generated SQL more generally, like
in the embedded server.

We also save and restore the SQL_LOG_BIN and WSREP_ON session server
variables so this can be included in the same session as other data
without taking into changes in state.

Remove wsrep.mysql_tzinfo_to_sql_symlink{,_skip} tests as they offered
no additional coverage beyond main.mysql_tzinfo_to_sql_symlink (no
server testing was done).

Add galera.mariadb_tzinfo_to_sql to actually test the replication
of tzinfo data through galera.

The conditional executable comment around /*M!100602 ...
START TRANSACTION .. LOCK TABLES.. */ is so that we can provide tzinfo
files (MDEV-27113, MDBF-389) and in the case that a user uses it on a
pre-10.6 server version it will still work. Both START TRANSACTION and
LOCK TABLES are not supported in prepared statements in MariaDB versions
earlier than 10.6.2.

Reviewed by Brandon Nesterenko
2022-04-21 14:59:29 +10:00