Commit graph

196798 commits

Author SHA1 Message Date
Vlad Lesin
090a84366a MDEV-29311 Server Status Innodb_row_lock_time% is reported in seconds
Before MDEV-24671, the wait time was derived from my_interval_timer() /
1000 (nanoseconds converted to microseconds, and not microseconds to
milliseconds like I must have assumed). The lock_sys.wait_time and
lock_sys.wait_time_max are already in milliseconds; we should not divide
them by 1000.

In MDEV-24738 the millisecond counts lock_sys.wait_time and
lock_sys.wait_time_max were changed to a 32-bit type. That would
overflow in 49.7 days. Keep using a 64-bit type for those millisecond
counters.

Reviewed by: Marko Mäkelä
2023-07-10 12:42:46 +03:00
Monty
6ed14bcc06 Disable view protocol for opt_trace.test
This is because some plan changes because of views
2023-07-07 12:53:50 +03:00
Monty
99bd226059 MDEV-31558 Add InnoDB engine information to the slow query log
The new statistics is enabled by adding the "engine", "innodb" or "full"
option to --log-slow-verbosity

Example output:

 # Pages_accessed: 184  Pages_read: 95  Pages_updated: 0  Old_rows_read: 1
 # Pages_read_time: 17.0204  Engine_time: 248.1297

Page_read_time is time doing physical reads inside a storage engine.
(Writes cannot be tracked as these are usually done in the background).
Engine_time is the time spent inside the storage engine for the full
duration of the read/write/update calls. It uses the same code as
'analyze statement' for calculating the time spent.

The engine statistics is done with a generic interface that should be
easy for any engine to use. It can also easily be extended to provide
even more statistics.

Currently only InnoDB has counters for Pages_% and Undo_% status.
Engine_time works for all engines.

Implementation details:

class ha_handler_stats holds all engine stats.  This class is included
in handler and THD classes.
While a query is running, all statistics is updated in the handler. In
close_thread_tables() the statistics is added to the THD.

handler::handler_stats is a pointer to where statistics should be
collected. This is set to point to handler::active_handler_stats if
stats are requested. If not, it is set to 0.
handler_stats has also an element, 'active' that is 1 if stats are
requested. This is to allow engines to avoid doing any 'if's while
updating the statistics.

Cloned or partition tables have the pointer set to the base table if
status are requested.

There is a small performance impact when using --log-slow-verbosity=engine:
- All engine calls in 'select' will be timed.
- IO calls for InnoDB reads will be timed.
- Incrementation of counters are done on local variables and accesses
  are inline, so these should have very little impact.
- Statistics has to be reset for each statement for the THD and each
  used handler. This is only 40 bytes, which should be neglectable.
- For partition tables we have to loop over all partitions to update
  the handler_status as part of table_init(). Can be optimized in the
  future to only do this is log-slow-verbosity changes. For this to work
  we have to update handler_status for all opened partitions and
  also for all partitions opened in the future.

Other things:
- Added options 'engine' and 'full' to log-slow-verbosity.
- Some of the new files in the test suite comes from Percona server, which
  has similar status information.
- buf_page_optimistic_get(): Do not increment any counter, since we are
  only validating a pointer, not performing any buf_pool.page_hash lookup.
- Added THD argument to save_explain_data_intern().
- Switched arguments for save_explain_.*_data() to have
  always THD first (generates better code as other functions also have THD
  first).
2023-07-07 12:53:18 +03:00
Marko Mäkelä
2855bc53bc Merge 10.5 into 10.6 2023-07-05 16:40:22 +03:00
Marko Mäkelä
bd7908e6ac MDEV-31568 InnoDB protection against dual processes accessing data insufficient
fil_node_open_file_low(): Always acquire an advisory lock on
the system tablespace. Originally, we already did this in
SysTablespace::open_file(), but SysTablespace::open_or_create()
would release those locks when it is closing the file handles.

This is a 10.5+ specific follow up to
commit 0ee1082bd2 (MDEV-28495).

Thanks to Daniel Black for verifying this bug.
2023-07-05 15:15:04 +03:00
Marko Mäkelä
5b62644e68 MDEV-31621 Remove ibuf_read_merge_pages() call from ibuf_insert_low()
When InnoDB attempts to buffer a change operation of a secondary index
leaf page (to insert, delete-mark or remove a record) and the
change buffer is too large, InnoDB used to trigger a change buffer merge
that could affect any tables. This could lead to huge variance in
system throughput and potentially unpredictable crashes, in case the
change buffer was corrupted and a crash occurred while attempting to
merge changes to a table that is not being accessed by the current
SQL statement.

ibuf_insert_low(): Simply return DB_STRONG_FAIL when the maximum size
of the change buffer is exceeded.

ibuf_contract_after_insert(): Remove.

ibuf_get_merge_page_nos_func(): Remove a constant parameter.
The function ibuf_contract() will be our only caller, during
shutdown with innodb_fast_shutdown=0.
2023-07-05 08:48:37 +03:00
Sergei Golubchik
46b79b8cd1 MDEV-30084 Shutdown hangs in some tests
debug-only issue. the test was doing

  set debug_sync='now SIGNAL go3';
  ...
  set debug_sync='reset';

which translated into

  add "go3" to the hash of active signals
  pthread_broadcast to wake up waiting threads
  ...
  clear the hash of active signals

as a result a waiting thread was awoken, but the hash was emptied
before the thread checked if its signal was in the hash. so the
thread didn't find its signal and went back to sleep.

let's wait until the awoken thread has completely finished
disconnecting and was added to the thread cache.
2023-07-04 16:37:30 +02:00
Sergei Golubchik
f6ecadfee8 fix ASAN+safemalloc builds
debug_sync refactoring introduced a statically instantiated object
debug_sync_global of the structure st_debug_sync_globals.
st_debug_sync_globals includes Hash_set<> which allocates memory
in the constructor. sf_malloc() calls _my_thread_var()->dbug_id
which is pthread_getspecific(THR_KEY_mysys), and THR_KEY_mysys is 0
before pthread_key_create(). pthread_getspecific(0) returns a valid
pointer, not EINVAL. And safemalloc dereferences it.

let's statically initialize THR_KEY_mysys to -1, this makes
pthread_getspecific(THR_KEY_mysys) to fail before pthread_key_create()
is called.

followup for 8885225de6
2023-07-04 16:37:29 +02:00
Sergei Golubchik
af38a8b438 cleanup: move Type_collection_fbt<> template out of Type_handler_fbt<> 2023-07-04 16:37:29 +02:00
Sergei Golubchik
c09b1583f5 cleanup: remove FixedBinTypeBundle
now the main class is the Type_handler_fbt.

It contains everything and generates objects of all other classes as
needed.
2023-07-04 16:37:29 +02:00
Sergei Golubchik
238cfcc729 cleanup: remove unused and unlinkable method
it was calling Fbt::to_string(char*, size_t)
which didn't exist
2023-07-04 16:37:29 +02:00
Sergei Golubchik
bbe44da240 cleanup: udt in sql_yac.yy 2023-07-04 16:37:29 +02:00
Sergei Golubchik
5bf80af06e cleanup: remove Type_collection::handler_by_name()
it's a hard-coded geometry-specific method, move it into
geometry-specific function. Will go away in the future.
2023-07-04 16:37:29 +02:00
Sergei Golubchik
a4817e1520 cleanup: String::strstr() const 2023-07-04 16:37:29 +02:00
Sergei Golubchik
f6e488cc6d MDEV-26506 Over-quoted JSON when combining JSON_ARRAYAGG with JSON_OBJECT
add the test case
2023-07-04 16:37:29 +02:00
Marko Mäkelä
cb364a78d6 MDEV-31619 dict_stats_persistent_storage_check() may show garbage during --bootstrap
dict_stats_persistent_storage_check(): Do not output errmsg
if opt_bootstrap holds, because the message buffer would likely
be uninitialized.
2023-07-04 15:24:57 +03:00
Daniel Black
5f2a77cef1 MDEV-31268 Fedora MariaDB-shared replaces mariadb-connector-c
file /usr/lib64/libmariadb.so.3 from install of MariaDB-shared-10.11.3-1.fc38.x86_64 conflicts with file from package mariadb-connector-c-3.3.5-1.fc38.x86_64
2023-07-04 09:01:01 +10:00
Marko Mäkelä
f7b8a2c953 MDEV-31607 ER_DUP_KEY in mysql.innodb_table_stats upon RENAME on sequence
ha_innobase::delete_table(): Also on DROP SEQUENCE, do try to drop any
persistent statistics. They should really not be created for
SEQUENCE objects (which internally are 1-row no-rollback tables),
but that is how happened to always work.
2023-07-03 16:47:58 +03:00
Marko Mäkelä
dc1bd1802a MDEV-31386 InnoDB: Failing assertion: page_type == i_s_page_type[page_type].type_value
i_s_innodb_buffer_page_get_info(): Correct a condition.
After crash recovery, there may be some buffer pool pages in FREED state,
containing garbage (invalid data page contents). Let us ignore such pages
in the INFORMATION_SCHEMA output.

The test innodb.innodb_defragment_fill_factor will be removed, because
the queries that it is invoking on information_schema.innodb_buffer_page
would start to fail. The defragmentation feature was removed in
commit 7ca89af6f8 in MariaDB Server 11.1.

Tested by: Matthias Leich
2023-07-03 14:39:29 +03:00
Marko Mäkelä
3d90143859 MDEV-31559 btr_search_hash_table_validate() does not check if CHECK TABLE is killed
btr_search_hash_table_validate(), btr_search_validate(): Add the
parameter THD for checking if the statement has been killed.
Any non-QUICK CHECK TABLE will validate the entire adaptive hash index
for all InnoDB tables, which may be extremely slow when running
multiple concurrent CHECK TABLE.
2023-06-30 17:07:21 +03:00
Oleg Smirnov
6d911219d6 MDEV-30639 Upgrade to 10.8 and later does not work on Windows
During the upgrade procedure on Windows mysqld.exe is started with
the named pipe connection protocol. mysqladmin.exe then pings the
server to check if is up and running. Command line looks like:
   mysqladmin.exe --protocol=pipe --socket=mysql_upgrade_service_xxx ping
But the "socket" parameter resets the "protocol" which was previously
initialized with the "pipe" value, setting it to "socket".
As a result, connection cannot be established and the upgrade
procedure fails.
"socket" in Windows is used to pass the name of the pipe so resetting
the protocol is not valid in this case.

This commit fixes resetting of the "protocol" parameter with "socket"
parameter in the case when protocol has been previously initialized
to "pipe" value
2023-06-30 15:59:14 +07:00
Alexander Barkov
fdab2c4c64 MDEV-31578 DECLARE CURSOR: "Memory not freed: 280 bytes lost" on syntax error
When CURSOR parameters get parsed, their sp_assignment_lex instances
(one instance per parameter) get collected to List<sp_assignment_lex>.

These instances get linked to sphead only in the end of the list.
If a syntax error happened in the middle of the parameter list,
these instances were not deleted, which caused memory leaks.

Fix:

using a Bison %destructor to free rules of the <sp_assignment_lex_list>
type (on syntax errors).

Afte the fix these sp_assignment_lex instances from CURSOR parameters
deleted as follows:

- If the CURSOR statement was fully parsed, then these instances
  get properly linked to sp_head structures, so they are deleted
  during ~sp_head (this did not change)

- If the CURSOR statement failed on a syntax error, then by Bison's
  %destructor (this is being added in the current patch).
2023-06-29 21:29:46 +04:00
Alexander Barkov
0d3720c12a MDEV-30680 Warning: Memory not freed: 280 on mangled query, LeakSanitizer: detected memory leaks
The parser works as follows:

The rule expr_lex returns a pointer to a newly created sp_expr_lex
instance which is not linked to any MariaDB structures yet - it is
pointed only from a Bison stack variable. The sp_expr_lex instance
gets linked to other structures (such as sp_instr_jump_if_not) later,
after scanning some following grammar.

Problem before the fix:
If a parse error happened immediately after expr_lex (before it got linked),
the created sp_expr_lex value got lost causing a memory leak.

Fix:

- Using Bison's "destructor" directive to free the results of expr_lex
  on parse/oom errors.

- Moving the call for LEX::cleanup_lex_after_parse_error() from
  MYSQL_YYABORT and yyerror inside parse_sql().
  This is needed because Bison calls destructors after yyerror(),
  while it's important to delete the sp_expr_lex instance before
  LEX::cleanup_lex_after_parse_error().
  The latter frees the memory root containing the sp_expr_lex instance.

  After this change the code block are executed in the following order:

  - yyerror() -- now only raises the error to DA (no cleanup done any more)
  - %destructor { delete $$; } <expr_lex>  -- destructs the sp_expr_lex instance
  - LEX::cleanup_lex_after_parse_error()   -- frees the memory root containing
                                              the sp_expr_lex instance

- Removing the "delete sublex" related code from restore_lex():
  - restore_lex() is called in most cases on success, when delete is not needed.
  - There is one place when restore_lex() is called on error:
    In sp_create_assignment_instr(). But in this case LEX::sp_lex_in_use
    is true anyway.
    The patch adds a new DBUG_ASSERT(lex->sp_lex_in_use) to guard this.
2023-06-29 13:34:22 +04:00
Marko Mäkelä
33877cfeae Fix WITH_UBSAN GCC -Wconversion 2023-06-28 17:07:00 +03:00
Vlad Lesin
3e89b4fcc6 MDEV-31570 gap_lock_split.test hangs sporadically
The fix is in replacing the waiting for the whole purge finishing
with the the waiting for only delete-marked records purging finishing.

Reviewed by: Marko Mäkelä
2023-06-28 14:22:40 +03:00
Vlad Lesin
687fd6bef5 MDEV-30648 btr_estimate_n_rows_in_range() accesses unfixed, unlatched page
The issue is caused by MDEV-30400 fix.

There are two cursors in btr_estimate_n_rows_in_range() - p1 and p2, but
both share the same mtr. Each cursor contains mtr savepoint for the
previously fetched block to release it then the current block is
fetched.

Before MDEV-30400 the block was released with
mtr_t::release_block_at_savepoint(), it just unfixed a block and
released its page patch. In MDEV-30400 it was replaced with
mtr_t::rollback_to_savepoint(), which does the same as the former
mtr_t::release_block_at_savepoint(ulint begin, ulint end) but also
erases the corresponding slots from mtr memo, what invalidates any
stored mtr's memo savepoints, greater or equal to "begin".

The idea of the fix is to get rid of savepoints at all in
btr_estimate_n_rows_in_range() and
btr_estimate_n_rows_in_range_on_level(). As
mtr_t::rollback_to_savepoint() erases elements from mtr_t::m_memo, we
know what element of mtr_t::m_memo can be deleted on the certain case,
so there is no need to store savepoints.

See also the following slides for details:
https://docs.google.com/presentation/d/1RFYBo7EUhM22ab3GOYctv3j_3yC0vHtBY9auObZec8U

Reviewed by: Marko Mäkelä
2023-06-28 11:00:02 +03:00
Yuchen Pei
24faa5de16
MDEV-30542 Fixing spider/bugfix.self_reference_multi
The server needs to have a unique name
2023-06-28 12:22:56 +10:00
Marko Mäkelä
694ce0d08e Merge 10.5 into 10.6 2023-06-27 13:03:32 +03:00
Marko Mäkelä
84dbd0253d MDEV-31487: Recovery or backup failure after innodb_undo_log_truncate=ON
recv_sys_t::parse(): For undo tablespace truncation mini-transactions,
remember the start_lsn instead of the end LSN. This is what we expect
after commit 461402a564 (MDEV-30479).
2023-06-27 09:12:38 +03:00
Marko Mäkelä
493083833b Merge 10.5 into 10.6 2023-06-26 17:11:38 +03:00
Rucha Deodhar
14275b4715 MDEV-28915: mysql_upgrade fails due to old_mode="", with "Cannot load from
mysql.proc. The table is probably corrupted"

Analysis: When mysql_upgrade runs statements for upgrade, characterset is
converted to utf8mb4 because server starts with old_mode that interprets
utf8 to utf8mb4, but mysql.proc table has "utf8mb3" as hardcoded, so
it crashes with corrupted table.

Fix: Changed Table_check_intact::check() definition to allow both
utf8mb3 and utf8mb4 by checking prefix and changing the upgrade scripts
to explicitly use utf8mb3
2023-06-26 15:10:30 +05:30
Monty
6daccd4e48 Moved test from group_min_max.test to group_min_max_not_embedded.test 2023-06-25 16:26:28 +03:00
Michael Widenius
3c7fd3c89b MDEV-23106 Unable to recognize/import partitioned tables from physical MySQL databases
MDEV-29253 Detect incompatible MySQL partition scheme and either convert
them or report to user and in error log.

This task is about converting in place MySQL 5.6 and 5.7 partition tables
to MariaDB as part of mariadb-upgrade.

- Update TABLE_SHARE::init_from_binary_frm_image() to be able to read
  MySQL frm files with partitions.
- Create .par file, if it do not exists, on open of partitioned table.

Executing mariadb-upgrade will create all the missing .par files.
The MySQL .frm file will be changed to MariaDB format after next
ALTER TABLE.

Other changes:
- If we are using stored mysql_version to distingush between MySQL and
  MariaDB  .frm file information, do not upgrade mysql_version in the
  .frm file as part of CHECK TABLE .. FOR UPGRADE as this would cause
  problems next time we parse the .frm file.
2023-06-25 16:15:08 +03:00
Monty
d671fec431 Fixed some errors & warnings when running mariadb-upgrade on MySQL instance
- Moved view checks after privilege tables are fixed. This is to avoid
  warnings about wrongly defined mysql.proc when checking views.
- Don't use stat tables before they have been fixed.
- Don't run mysql_fix_view() if 'FOR MYSQL' is used if the view is
  already a MariaDB view.
- Added 'FOR UPGRADE' as an option for 'REPAIR VIEW' to be able to
  detect if the REPAIR command comes from mariadb_upgrade. In this
  case we get a warning, instead of an error, if a definer of a view
  does not exists.
2023-06-25 16:15:08 +03:00
Vicentiu Ciorbaru
324d8a6036 Fix segfault in find_files if db is NULL and path does not exist or can't be read
In this case, one would end up calling my_error(ER_BAD_DB_ERROR, ...)
with a null ptr for DB, which caused a NULL ptr dereference.
2023-06-25 16:15:08 +03:00
Vicentiu Ciorbaru
38fe266ea9 Fix gcc warning for wsrep_plug 2023-06-25 16:15:08 +03:00
Vicentiu Ciorbaru
f20d556f19 Fix use of uninitialized variable
The original code generated a warning in gcc 13.1
2023-06-25 16:15:08 +03:00
Monty
ecd8a52252 Renamed 'mysql' -> 'mariadb' in mtr 2023-06-25 16:15:08 +03:00
Brandon Nesterenko
c2d44ecb90 MDEV-29894: Calling a function from a different database in a slave side trigger crashes
When opening and locking tables, if triggers will be invoked in a
separate database, thd->set_db() is invoked, thus freeeing the memory
and headers which thd->db had previously pointed to. In row based
replication, the event execution logic initializes thd->db to point
to the database which the event targets, which is owned by the
corresponding table share (introduced in d9898c9 for MDEV-7409).
The problem then, is that during the table opening and locking
process for a row event, memory which belongs to the table share
would be freed, which is not valid.

This patch replaces the thd->reset_db() calls to thd->set_db(),
which copies-by-value, rather than by reference. Then when the
memory is freed, our copy of memory is freed, rather than memory
which belongs to a table share.

Notes:
  1. The call to change thd->db now happens on a higher-level, in
Rows_log_event::do_apply_event() rather than ::do_exec_row(), in the
call stack. This is because do_exec_row() is called within a loop,
and each invocation would redundantly set and unset the db to the
same value.
  2. thd->set_db() is only used if triggers are to be invoked, as
there is no vulnerability in the non-trigger case, and copying
memory would be an unnecessary inefficiency.

Reviewed By:
============
Andrei Elkin <andrei.elkin@mariadb.com>
2023-06-21 12:51:01 -06:00
Sergei Petrunia
0b61f4e0e4 Fix comment 2023-06-20 12:58:39 +03:00
Thirunarayanan Balathandayuthapani
bd076d4dff MDEV-31442 page_cleaner thread aborts while releasing the tablespace
- InnoDB shouldn't acquire the tablespace when it is being stopped
or closed
2023-06-16 14:58:48 +05:30
Sergei Petrunia
f7e9ac0d88 MDEV-31449: Assertion s->table->opt_range_condition_rows <= s->found_records
Fix a typo in make_join_statistics(): when updating statistics for
derived table, set s->table->... not "table->..."
2023-06-15 11:27:31 +03:00
Sergei Petrunia
0e2e70c4c1 MDEV-31479: Inconsistency between MRR and SQL layer costs can cause poor query plan
(Same as
TODO-3938: best_access_path shows negative costs for mrr=on)

best_access_path() assumes that quick select cost includes
(quick->rows/TIME_FOR_COMPARE) as a cost of checking the attached
part of the WHERE condition.

It calls adjust_quick_cost() to subtract addition from quick's cost.

The problem was that DS-MRR cost formula didn't include this cost.
For very large tables, adjust_quick_cost() would produce a negative
cost which would cause assert in debug build or bad query plan choice
in release builds.

Approved-by: Monty <monty@mariadb.org>
2023-06-14 13:56:33 +03:00
Thirunarayanan Balathandayuthapani
841e905f20 MDEV-31442 page_cleaner thread aborts while releasing the tablespace
After further I/O on a tablespace has been stopped
(for example due to DROP TABLE or an operation that
rebuilds a table), page cleaner thread tries to
flush the pending writes for the tablespace and
releases the tablespace reference even though it was not
acquired.

fil_space_t::flush(): Don't release the tablespace when it is
being stopped and closed

Thanks to Marko Mäkelä for suggesting this patch.
2023-06-09 18:15:33 +05:30
Tuukka Pasanen
8171f9da87 MDEV-31423: Make sure that datadir is available with SySV-init script
Commit fixes Debian SySV-init script fail:
  /etc/init.d/mariadb: line 90: [: : integer expression expected
which happens if datadir is not changed in configuration which
makes it invisible when printing MariaDB config defaults.

Commit makes sure that there is some value if nothing else if in hand
use default /usr/lib/mysql or fail with correct error message if
directory is not present
2023-06-09 10:22:34 +10:00
Thirunarayanan Balathandayuthapani
bf0a54df34 MDEV-31416 ASAN errors in dict_v_col_t::detach upon adding key to virtual column
- InnoDB throws ASAN error while adding the index on virtual column
of system versioned table. InnoDB wrongly assumes that virtual
column collation type changes, creates new column with different
character set. This leads to failure while detaching the column
from indexes.
2023-06-08 16:34:45 +05:30
Sergei Petrunia
b3074128a6 MDEV-31380 post-fix: fix group_min_max.test with embedded and view-protocol
embedded doesn't have optimizer trace,
view-protocol doesn't work with long column names.
2023-06-08 11:14:50 +03:00
Marko Mäkelä
80585c9d6f Merge 10.5 into 10.6 2023-06-08 10:42:56 +03:00
Marko Mäkelä
d3eefbaa55 MDEV-31355 fixup: Adjust one more test
The test gcol.gcol_purge would reliably hang on 10.6 on a
Microsoft Windows builder without this adjustment.
A similar adjustment was applied in
commit 3e40f9a7f3
to the tests innodb.dml_purge and innodb.instant_alter_purge.
2023-06-08 10:40:48 +03:00
Marko Mäkelä
21031b24fc Suppress an occasional buffer pool warning 2023-06-08 09:38:03 +03:00