In commit 5ea5291 @sanja-byelkin for unknown reason switched the file mode
for 3 Galera tzinfo related test files from 644 -> 755. This exists only
from branch 10.6 onward:
$ git checkout 10.5
$ find mysql-test -executable -name *.test -or -executable -name *.result
(no results)
$ git checkout 10.6
$ find mysql-test -executable -name *.test -or -executable -name *.result
mysql-test/suite/galera/t/mysql_tzmysql-test/suite/galera/t/mysql_tzinfo_to_sql.test
mysql-test/suite/galera/t/mariadb_tzinfo_to_sql.test
mysql-test/suite/galera/r/mariadb_tzinfo_to_sql.resultinfo_to_sql.test
mysql-test/suite/galera/t/mariadb_tzinfo_to_sql.test
mysql-test/suite/galera/r/mariadb_tzinfo_to_sql.result
No test file nor test result file should be executable, so run chmod -x
on them.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
copy_back(): Also copy the dummy empty ib_logfile0 so that
MariaDB Server 10.8 or later can be started after
--copy-back or --move-back.
Thanks to Daniel Black for reporting this.
This is a 10.5 version of
commit ebf3649259
Add threadpool functionality to restrict concurrency during "batch"
periods (where tasks are added in rapid succession).
This will throttle thread creation more agressively than usual, while
keeping performance at least on-par.
One of these cases is bufferpool load, where async read IOs are executed
without any throttling. There can be as much as 650K read IOs for
loading 10GB buffer pool.
Another one is recovery, where "fake read" IOs are executed.
Why there are more threads than we expect?
Worker threads are not be recognized as idle, until they return to the
standby list, and to return to that list, they need to acquire
mutex currently held in the submit_task(). In those cases, submit_task()
has no worker to wake, and would create threads until default concurrency
level (2*ncpus) is satisfied. Only after that throttling would happen.
The problem was that we did not handle errors properly in
JOIN::get_best_combination. In case an early error, JOIN->join_tab would
contain unintialized values, which would cause errors on cleanup().
The error in question was reported earlier, but not noticed until later.
One cause of this is that most of the sql_select.cc code just checks
thd->fatal_error and not thd->is_error().
Fixed by changing of checks of fatal_error to is_error().
This allows a user to to change the default value of MAX_SEL_ARGS (16000)
in the rare case where they neeed more generated SEL_ARGS (as part of
the range optimizer)
Raise notes if indexes cannot be used:
- in case of data type or collation mismatch (diferent error messages).
- in case if a table field was replaced to something else
(e.g. Item_func_conv_charset) during a condition rewrite.
Added option to write warnings and notes to the slow query log for
slow queries.
New variables added/changed:
- note_verbosity, with is a set of the following options:
basic - All old notes
unusable_keys - Print warnings about keys that cannot be used
for select, delete or update.
explain - Print unusable_keys warnings for EXPLAIN querys.
The default is 'basic,explain'. This means that for old installations
the only notable new behavior is that one will get notes about
unusable keys when one does an EXPLAIN for a query. One can turn all
of all notes by either setting note_verbosity to "" or setting sql_notes=0.
- log_slow_verbosity has a new option 'warnings'. If this is set
then warnings and notes generated are printed in the slow query log
(up to log_slow_max_warnings times per statement).
- log_slow_max_warnings - Max number of warnings written to
slow query log.
Other things:
- One can now use =ALL for any 'set' variable to set all options at once.
For example using "note_verbosity=ALL" in a config file or
"SET @@note_verbosity=ALL' in SQL.
- mysqldump will in the future use @@note_verbosity=""' instead of
@sql_notes=0 to disable notes.
- Added "enum class Data_type_compatibility" and changing the return type
of all Field::can_optimize*() methods from "bool" to this new data type.
Reviewer & Co-author: Alexander Barkov <bar@mariadb.com>
- The code that prints out the notes comes mainly from Alexander
The warning is given in case of table not found or if there is a lock
timeout. The warning is needed as in case of a lock timeout then the
persistent table stats are going to be wrong.
Updated ha_mroonga::storage_check_if_supported_inplace_alter to support
new ALTER TABLE flags.
This fixes failing tests:
mroonga/storage.alter_table_add_index_unique_duplicated
mroonga/storage.alter_table_add_index_unique_multiple_column_duplicated
Example of what causes the problem:
T1: ANALYZE TABLE starts to collect statistics
T2: ALTER TABLE starts by deleting statistics for all changed fields,
then creates a temp table and copies data to it.
T1: ANALYZE ends and writes to the statistics tables.
T2: ALTER TABLE renames temp table in place of the old table.
Now the statistics from analyze matches the old deleted tables.
Fixed by waiting to delete old statistics until ALTER TABLE is
the only one using the old table and ensure that rename of columns
can handle swapping of column names.
rename_columns_in_stat_table() (former rename_column_in_stat_tables())
now takes a list of columns to rename. It uses the following algorithm
to update column_stats to be able to handle circular renames
- While there are columns to be renamed and it is the first loop or
last rename loop did change something.
- Loop over all columns to be renamed
- Change column name in column_stat
- If fail because of duplicate key
- If this is first change attempt for this column
- Change column name to a temporary column name
- If there was a conflicting row, replace it with the current row.
else
- Remove entry from column list
- Loop over all remaining columns in the list
- Remove the conflicting row
- Change column from temporary name to final name in column_stat
Other things:
- Don't flush tables for every operation. Only flush when all updates
are done.
- Rename of columns was not handled in case of ALGORITHM=copy (old bug).
- Fixed that we do not collect statistics for hidden hash columns
used by UNIQUE constraint on long values.
- Fixed that we do not collect statistics for blob columns referred by
generated virtual columns. This was achieved by storing the fields for
which we want to have statistics in table->has_value_set instead of
in table->read_set.
- Rename of indexes was not handled for persistent statistics.
- This is now handled similar as rename of columns. Renamed columns
are now stored in 'rename_stat_indexes' and handled in
Alter_info::delete_statistics() together with drooped indexes.
- ALTER TABLE .. ADD INDEX may instead of creating a new index rename
an existing generated foreign key index. This was not reflected in
the index_stats table because this was handled in
mysql_prepare_create_table instead instead of in the mysql_alter() code.
Fixed by adding a call in mysql_prepare_create_table() to drop the
changed index.
I also had to change the code that 'marked the index' to be ignored
with code that would not destroy the original index name.
Reviewer: Sergei Petrunia <sergey@mariadb.com>
In `pseudo_slave_mode=1` aka "psedo-slave" mode any prepared XA
transaction disconnects from the user session, as if the user
connection drops. The xid of such transaction remains in the server,
and should the prepared transaction be read-only, it is marked.
The marking makes sure that the following termination of the
read-only transaction ends up with ER_XA_RBROLLBACK.
This did not take place actually for `pseudo_slave_mode=1` read-only.
Fixed with checking the read-only status of a prepared transaction
at time it disconnects from the `pseudo_slave_mode=1` session, to mark
its xid when that's the case.
Problem was that brute force (BF) thread requested conflicting lock
and was trying to kill victim transaction, but this victim
was also brute force thread. However, this victim was not actually
holding conflicting lock, instead both brute force transaction
and victim transaction were had insert intention locks.
We should not kill brute force victim transaction if requesting
lock does not need to wait.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
Spider is part of the server, and there's no need to check the
version.
All spider plugins are uninstalled in clean_up_spider.inc
DROP SERVER IF EXISTS makes things easier
is_file_on_ssd() is more expensive than it should be.
It caches the results by volume name, but still calls GetVolumePathName()
every time, which, as procmon shows, opens multiple directories in
filesystem hierarchy (db directory, datadir, and all ancestors)
The fix is to cache SSD status by volume serial ID, which is cheap to
retrieve with GetFileInformationByHandleEx()
Follow-up to fix issue with access to probably not-initialized mutex/cond_var
Constructor of the class st_debug_sync_globals was changed to initialize
the data members dsp_hits, dsp_executed, dsp_max_active with zero.
Formerly, these data members were filled with zeroes by C-runtime since
the variable debug_sync_global was declared as static and according with C rules
the static variable initialized with zero bytes.
By the same reason, the data members
debug_sync_global->ds_mutex
debug_sync_global->ds_cond
were initialized by zeros before the patch for MDEV-31871. After this patch
the memory for the synch primitives debug_sync_global->ds_mutex
and debug_sync_global->ds_cond are initialized explicitly by calling
the functions mysql_mutex_init/mysql_cond_init so access to these synch
primitives should be done only after such initialization be completed.
Guarded access to these synch primitives has been added to the function
debug_sync_end_thread() that is called on clean up since that was single
problem place detected by MSAN. Theoretically problem places located in the
function debug_sync_execute were not protected with similar check since
it is not obvious that the variables debug_sync_global->ds_mutex
and debug_sync_global->ds_cond could be not initilialized for use cases where
the function debug_sync_execute() is called. It is required additional study
to conclude whether it does need or not.
mariadb-install crashes on start when the static variable debug_sync_global
of the class st_debug_sync_globals is initialized by constructor.
Definition of the class st_debug_sync_globals has a data member of the type
Hash_set whose implementation depends on thread-specific data associated
with the key THD_KEY_mysys. This dependency results from constructor of
the class Hash_set that runs my_hash_init2() which in turn invokes my_malloc.
The thread-specific data value associated with the key THD_KEY_mysys is used
by the function sf_malloc to get id of the current thread. The key
THD_KEY_mysys is defined as static variable at my_thr_init.c initialized
with the value -1. Proper initialization of the key THD_KEY_mysys is done
with the library call pthread_key_create but it happens at the my_init()
that called much later after the first time the THD_KEY_mysys() has been
invoked. In according with Single Unix Specification, the effect of calling
pthread_setspecific() or pthread_getspecific() with a key value not obtained
from pthread_key_create() is undefined. That is the reason why mariadb-install
crashes on MacOS.
To fix the issue, the static variable debug_sync_global is converted to
a pointer to the class st_debug_sync_globals and its instantiation is done
explicitly at the function debug_sync_init() that is called at right time.
This is the follow-up patch to the commits
8885225de6f6ecadfee8
where was introduced a statically instantiated object debug_sync_global of
the structure st_debug_sync_globals and the key THR_KEY_mysys for this
thread-specific data was initialized with the value -1.
trx_t::set_skip_lock_inheritance() must be invoked at the very beginning
of lock_release_on_prepare(). Currently trx_t::set_skip_lock_inheritance()
is invoked at the end of lock_release_on_prepare() when lock_sys and trx
are released, and there can be a case when locks on prepare are released,
but "not inherit gap locks" bit has not yet been set, and page split
inherits lock to supremum.
Also reset supremum bit and rebuild waiting queue when XA is prepared.
Reviewed by: Marko Mäkelä
When resolving a column from the HAVING clause, a new Item_field
object may be created inside Item_ref::fix_fields().
But the object is created with an empty name resolution context,
which then leads to debug assertion failure during
Item_field::fix_fields().
The solution is to pass the correct name resolution context
when creating the Item_field object.
Reviewer: Oleksandr Byelkin (sanja@mariadb.com)
The table structure from MySQL-5.1.14 is:
CREATE TABLE `slow_log` (
`start_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`user_host` mediumtext NOT NULL,
`query_time` time NOT NULL,
`lock_time` time NOT NULL,
`rows_sent` int(11) NOT NULL,
`rows_examined` int(11) NOT NULL,
`db` varchar(512) DEFAULT NULL,
`last_insert_id` int(11) DEFAULT NULL,
`insert_id` int(11) DEFAULT NULL,
`server_id` int(11) DEFAULT NULL,
`sql_text` mediumtext NOT NULL
) ENGINE=CSV DEFAULT CHARSET=utf8 COMMENT='Slow log'
Even as far back as MySQL-5.5.40 this table could be created as NULLs
where not permitted in the CSV table time, but it seems they
where allowed sometime.
As the first part of mariadb-upgrade adds the column thread_id without
correcting the 'NULL'able status of existing columns it fails.
We reorder the sql statements in the ugprade as follows:
ALTER TABLE slow_log MODIFY {columns} {new types} NOT NULL,....
As thread_id doesn't exist in the above statement it was removed from
the first ALTER TABLE statement to prevent failure.
Previous ALTER TABLE slow_log where moved later appending thread_id
and rows_affected, and also enforces the type of thread_id if it
was incorrectly like the now first ALTER STATEMENT slow_log used
to do.
Updated ha_mroonga::storage_check_if_supported_inplace_alter to support
new ALTER TABLE flags.
This fixes failing tests:
mroonga/storage.alter_table_add_index_unique_duplicated
mroonga/storage.alter_table_add_index_unique_multiple_column_duplicated
Problem:
=======
- There is a race condition between purge and rollback of alter
operation. Alter rollback marks the index as corrupted.
At the same time, purge is working on the same index and leads
to assert failure. This is caused by
commit 7c0b9c6020 (MDEV-15250).
Solution:
=======
- After MDEV-15250, InnoDB logs the operation only at the
end of transaction commit and applies the log in
ha_innobase::commit_inplace_alter_table() and
also via dml thread. So there is no need for purge to work
on uncommitted index.
The assertion would fail in the test innodb.innodb-index-online
when the following call is added to the start of the function
row_purge_remove_sec_if_poss_leaf():
if (!index->is_committed()) sleep(5);
According to commit ea56841997
the stack normally grows downwards, except on HP PA-RISC where
it grows upwards. Because determining the stack direction is not
possible in a portable way, let us determine the default STACK_DIRECTION
in CMake based on the ISA.
On clang 16.0.6 running on and targeting AMD64, STACK_DIRECTION=1 is
being incorrectly detected, causing the failure of a number of tests.
row_vers_vc_matches_cluster(): Invoke dtype_get_at_most_n_mbchars()
to extract the correct number of bytes corresponding to the number
of characters in a virtual column prefix index, just like we do in
row_sel_sec_rec_is_for_clust_rec().
The test case would occasionally reproduce the failure when this
fix is not present.
On creation of a VIEW that depends on a stored routine an instance of
the class Item_func_sp is allocated on a memory root of SP statement.
It happens since mysql_make_view() calls the method
THD::activate_stmt_arena_if_needed()
before parsing definition of the view.
On the other hand, when sp_head's rcontext is created an instance of
the class Field referenced by the data member
Item_func_sp::result_field
is allocated on the Item_func_sp's Query_arena (call arena) that set up
inside the method
Item_sp::execute_impl
just before calling the method
sp_head::execute_function()
On return from the method sp_head::execute_function() all items allocated
on the Item_func_sp's Query_arena are released and its memory root is freed
(see implementation of the method Item_sp::execute_impl). As a consequence,
the pointer
Item_func_sp::result_field
references to the deallocated memory. Later, when the method
sp_head::execute
cleans up items allocated for just executed SP instruction the method
Item_func_sp::cleanup is invoked and tries to delete an object referenced
by data member Item_func_sp::result_field that points to already deallocated
memory, that results in a server abnormal termination.
To fix the issue the current active arena shouldn't be switched to
a statement arena inside the function mysql_make_view() that invoked indirectly
by the method sp_head::rcontext_create. It is implemented by introducing
the new Query_arena's state STMT_SP_QUERY_ARGUMENTS that is set when explicit
Query_arena is created for placing SP arguments and other caller's side items
used during SP execution. Then the method THD::activate_stmt_arena_if_needed()
checks Query_arena's state and returns immediately without switching to
statement's arena.
mariadb-install-db --auth-root-authentication-method=normal created 4
root accounts by default, but only two of these had PROXY privilege
granted.
mariadb-install-db (default option
--auth-root-authentication-method=socket) as non-root user also didn't
grant PROXY priv to the created nonroot@localhost user.
To fix this, in mysql_system_tables_data.sql, we re-use tmp_user_nopasswd
as this contains the list of all root users.
REPLACE INTO tmp_proxies_priv SELECT @current_hostname, IFNULL(@auth_root_socket, 'root')
creates the $user@$current_host but will not error if @auth_root_socket
is null. Note @current_hostname lines are filtered out with
--cross-bootstrap in mariadb-install-db so it was needed to include this
expression for consistency.
Like the existing mysql_system_tables.sql is used to create teh
$user@localhost proxies_priv.
Test cases roles.acl_statistics, perfschema,privilege_table_io depends on the number of proxy users.
After:
--auth-root-authentication-method=normal:
MariaDB [mysql]> select * from global_priv;
+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------+
| Host | User | Priv |
+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------+
| localhost | mariadb.sys | {"access":0,"plugin":"mysql_native_password","authentication_string":"","account_locked":true,"password_last_changed":0} |
| localhost | root | {"access":18446744073709551615} |
| bark | root | {"access":18446744073709551615} |
| 127.0.0.1 | root | {"access":18446744073709551615} |
| ::1 | root | {"access":18446744073709551615} |
| localhost | | {} |
| bark | | {} |
+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------+
7 rows in set (0.001 sec)
MariaDB [mysql]> select * from proxies_priv;
+-----------+------+--------------+--------------+------------+---------+---------------------+
| Host | User | Proxied_host | Proxied_user | With_grant | Grantor | Timestamp |
+-----------+------+--------------+--------------+------------+---------+---------------------+
| localhost | root | | | 1 | | 2023-07-10 12:12:24 |
| 127.0.0.1 | root | | | 1 | | 2023-07-10 12:12:24 |
| ::1 | root | | | 1 | | 2023-07-10 12:12:24 |
| bark | root | | | 1 | | 2023-07-10 12:12:24 |
+-----------+------+--------------+--------------+------------+---------+---------------------+
--auth-root-authentication-method=socket:
MariaDB [mysql]> select * from proxies_priv;
+-----------+------+--------------+--------------+------------+---------+---------------------+
| Host | User | Proxied_host | Proxied_user | With_grant | Grantor | Timestamp |
+-----------+------+--------------+--------------+------------+---------+---------------------+
| localhost | root | | | 1 | | 2023-07-10 12:11:55 |
| localhost | dan | | | 1 | | 2023-07-10 12:11:55 |
| bark | dan | | | 1 | | 2023-07-10 12:11:55 |
+-----------+------+--------------+--------------+------------+---------+---------------------+
3 rows in set (0.017 sec)
MariaDB [mysql]> select * from global_priv;
+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------------------+
| Host | User | Priv |
+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------------------+
| localhost | mariadb.sys | {"access":0,"plugin":"mysql_native_password","authentication_string":"","account_locked":true,"password_last_changed":0} |
| localhost | root | {"access":18446744073709551615,"plugin":"mysql_native_password","authentication_string":"invalid","auth_or":[{},{"plugin":"unix_socket"}]} |
| localhost | dan | {"access":18446744073709551615,"plugin":"mysql_native_password","authentication_string":"invalid","auth_or":[{},{"plugin":"unix_socket"}]} |
| localhost | | {} |
| bark | | {} |
+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------------------+
5 rows in set (0.000 sec)
MariaDB [mysql]> show grants;
+----------------------------------------------------------------------------------------------------------------------------------------+
| Grants for dan@localhost |
+----------------------------------------------------------------------------------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO `dan`@`localhost` IDENTIFIED VIA mysql_native_password USING 'invalid' OR unix_socket WITH GRANT OPTION |
| GRANT PROXY ON ''@'%' TO 'dan'@'localhost' WITH GRANT OPTION |
+----------------------------------------------------------------------------------------------------------------------------------------+
