Consider a query of the form:
select ... from (select item2 as COL1) as T where COL1=123
Condition pushdown into derived table will try to push "COL1=123" condition
down into table T.
The process of pushdown involves "substituting" the item, that is,
replacing Item_field("T.COL1") with its "producing item" item2.
In order to use item2, one needs to clone it (call Item::build_clone).
If the item is not cloneable (e.g. Item_func_sp is not), the pushdown
process will fail and nothing at all will be pushed.
Fixed by introducing transform_condition_or_part() which will try to apply
the transformation for as many parts of condition as possible. The parts of
condition that couldn't be transformed are dropped.
Add KEYWORDS table and SQL_FUNCTIONS table to INFORMATION_SCHEMA.
This commits needs some minor changes when propagated upwards
(e.g. func_array in item_create.cc has a termination element that
doesn't exist in later versions of MariaDB)
If the first token of the body of a stored procedure was 'WITH' then
the beginning of the body was determined incorrectly and that token was
missing in the string representing the body of the SP in mysql.proc. As a
resultnany call of such procedure failed as the string representing the
body could not be parsed.
The patch corrects the code of the functions get_tok_start() and
get_cpp_tok_start() of the class Lex_input_stream to make them take into
account look ahead tokens. The patch is needed only for 10.2 as this
problem has neen resolved in 10.3+.
wsrep_sst_common did not correctly set name for binlog index
file if custom binlog name was used and this name was
not added to script command line.
Added test case for both log_basename and log_binlog.
wsrep_sst_common did not correctly set name for binlog index
file if custom binlog name was used and this name was
not added to script command line.
Added test case for both log_basename and log_binlog.
This bug affected queries with views / derived_tables / CTEs whose
specifications were of the form
(SELECT ... LIMIT <n>) ORDER BY ...
Units representing such specifications contains one SELECT_LEX structure
for (SELECT ... LIMIT <n>) and additionally SELECT_LEX structure for
fake_select_lex. This fact should have been taken into account in the
function mysql_derived_fill().
This patch has to be applied to 10.2 and 10.3 only.
This bug affected queries with views / derived_tables / CTEs whose
specifications were of the form
(SELECT ... LIMIT <n>) ORDER BY ...
Units representing such specifications contains one SELECT_LEX structure
for (SELECT ... LIMIT <n>) and additionally SELECT_LEX structure for
fake_select_lex. This fact should have been taken into account in the
function mysql_derived_fill().
This patch has to be applied to 10.2 and 10.3 only.
- InnoDB fails to check DB_COMPUTE_VALUE_FAILED error in
row_merge_read_clustered_index() and wrongly asserts that
the buffer shouldn't be ran out of memory. Alter table
should give warning when the column value is being
truncated.
* Make Item_in_optimizer::fix_fields inherit the with_window_func
attribute of the subquery's left expression (the subquery itself
cannot have window functions that are aggregated in this select)
* Make Item_cache_wrapper::Item_cache_wrapper() inherit
with_window_func attribute of the item it is caching.
If a select query contained an ORDER BY clause that followed a LIMIT clause
or an ORDER BY clause or ORDER BY with LIMIT the EXPLAIN output for the
query showed an execution plan different from that was actually executed.
Approved by Roman Nozdrin <roman.nozdrin@mariadb.com>
If a join query uses a derived table (view / CTE) with GROUP BY clause then
the execution plan for such join may employ split optimization. When this
optimization is employed the derived table is not materialized. Rather only
some partitions of the derived table are subject to grouping. Split
optimization can be applied only if:
- there are some indexes over the tables used in the join specifying the
derived table whose prefixes partially cover the field items used in the
GROUP BY list (such indexes are called splitting indexes)
- the WHERE condition of the join query contains conjunctive equalities
between columns of the derived table that comprise major parts of
splitting indexes and columns of the other join tables.
When the optimizer evaluates extending of a partial join by the rows of the
derived table it always considers a possibility of using split optimization.
Different splitting indexes can be used depending on the extended partial
join. At some rare conditions, for example, when there is a non-splitting
covering index for a table joined in the join specifying the derived table
usage of a splitting index to produce rows needed for grouping may be still
less beneficial than usage of such covering index without any splitting
technique. The function JOIN_TAB::choose_best_splitting() must take this
into account.
Approved by Oleksandr Byelkin <sanja@mariadb.com>
This bug could manifest itself after pushing a where condition over a
mergeable derived table / view / CTE DT into a grouping view / derived
table / CTE V whose item list contained set functions with constant
arguments such as MIN(2), SUM(1) etc. In such cases the field references
used in the condition pushed into the view V that correspond set functions
are wrapped into Item_direct_view_ref wrappers. Due to a wrong implementation
of the virtual method const_item() for the class Item_direct_view_ref the
wrapped set functions with constant arguments could be erroneously taken
for constant items. This could lead to a wrong result set returned by the
main select query in 10.2. In 10.4 where a possibility of pushing condition
from HAVING into WHERE had been added this could cause a crash.
Approved by Sergey Petrunya <sergey.petrunya@mariadb.com>
Both EXPLAIN and EXPLAIN EXTENDED statements produce different results set
in case it is run in normal way and in PS mode for the statements
UPDATE/DELETE with subquery.
The use case below reproduces the issue:
MariaDB [test]> CREATE TABLE t1 (c1 INT KEY) ENGINE=MyISAM;
Query OK, 0 rows affected (0,128 sec)
MariaDB [test]> CREATE TABLE t2 (c2 INT) ENGINE=MyISAM;
Query OK, 0 rows affected (0,023 sec)
MariaDB [test]> CREATE TABLE t3 (c3 INT) ENGINE=MyISAM;
Query OK, 0 rows affected (0,021 sec)
MariaDB [test]> EXPLAIN EXTENDED UPDATE t3 SET c3 =
-> ( SELECT COUNT(d1.c1) FROM ( SELECT a11.c1 FROM t1 AS a11
-> STRAIGHT_JOIN t2 AS a21 ON a21.c2 = a11.c1 JOIN t1 AS a12
-> ON a12.c1 = a11.c1 ) d1 );
+------+-------------+-------+------+---------------+------+---------+------+------+----------+--------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+------+-------------+-------+------+---------------+------+---------+------+------+----------+--------------------------------+
| 1 | PRIMARY | t3 | ALL | NULL | NULL | NULL | NULL | 0 | 100.00 | |
| 2 | SUBQUERY | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | Impossible WHERE noticed after reading const tables
+------+-------------+-------+------+---------------+------+---------+------+------+----------+--------------------------------+
2 rows in set (0,002 sec)
MariaDB [test]> PREPARE stmt FROM
-> EXPLAIN EXTENDED UPDATE t3 SET c3 =
-> ( SELECT COUNT(d1.c1) FROM ( SELECT a11.c1 FROM t1 AS a11
-> STRAIGHT_JOIN t2 AS a21 ON a21.c2 = a11.c1 JOIN t1 AS a12
-> ON a12.c1 = a11.c1 ) d1 );
Query OK, 0 rows affected (0,000 sec)
Statement prepared
MariaDB [test]> EXECUTE stmt;
+------+-------------+-------+------+---------------+------+---------+------+------+----------+--------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+------+-------------+-------+------+---------------+------+---------+------+------+----------+--------------------------------+
| 1 | PRIMARY | t3 | ALL | NULL | NULL | NULL | NULL | 0 | 100.00 | |
| 2 | SUBQUERY | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | no matching row in const table |
+------+-------------+-------+------+---------------+------+---------+------+------+----------+--------------------------------+
2 rows in set (0,000 sec)
The reason by that different result sets are produced is that on execution
of the statement 'EXECUTE stmt' the flag SELECT_DESCRIBE not set
in the data member SELECT_LEX::options for instances of SELECT_LEX that
correspond to subqueries used in the UPDTAE/DELETE statements.
Initially, these flags were set on parsing the statement
PREPARE stmt FROM "EXPLAIN EXTENDED UPDATE t3 SET ..."
but latter they were reset before starting real execution of
the parsed query during handling the statement 'EXECUTE stmt';
So, to fix the issue the functions mysql_update()/mysql_delete()
have been modified to set the flag SELECT_DESCRIBE forcibly
in the data member SELECT_LEX::options for the primary SELECT_LEX
of the UPDATE/DELETE statement.
The only call of the virtual member function
handler::update_table_comment() was removed in
commit 82d28fada7 (MySQL 5.5.53)
but the implementation was not removed.
The only non-trivial implementation was for InnoDB. The information
is now returned via handler::get_foreign_key_create_info() and
ha_statistics::delete_length.
The problem was that when LOCK TABLES where unwinded as part of
a killed connection, unlink_all_closed_tables() did not like that
there was uncommited transactions.
Fixed by doing a rollback of any open transaction in this particular case.
cherry-pick commit: 1fff2398ef
MDEV-22530 post push fixes from 10.6.
Followup. If the KILL happens - report it as a failure,
don't eat it up silently. Note that this has to be done after `table_name`
is populated, so that the error message could show it.
In the code existed just before this patch binding of a table reference to
the specification of the corresponding CTE happens in the function
open_and_process_table(). If the table reference is not the first in the
query the specification is cloned in the same way as the specification of
a view is cloned for any reference of the view. This works fine for
standalone queries, but does not work for stored procedures / functions
for the following reason.
When the first call of a stored procedure/ function SP is processed the
body of SP is parsed. When a query of SP is parsed the info on each
encountered table reference is put into a TABLE_LIST object linked into
a global chain associated with the query. When parsing of the query is
finished the basic info on the table references from this chain except
table references to derived tables and information schema tables is put
in one hash table associated with SP. When parsing of the body of SP is
finished this hash table is used to construct TABLE_LIST objects for all
table references mentioned in SP and link them into the list of such
objects passed to a pre-locking process that calls open_and_process_table()
for each table from the list.
When a TABLE_LIST for a view is encountered the view is opened and its
specification is parsed. For any table reference occurred in
the specification a new TABLE_LIST object is created to be included into
the list for pre-locking. After all objects in the pre-locking have been
looked through the tables mentioned in the list are locked. Note that the
objects referenced CTEs are just skipped here as it is impossible to
resolve these references without any info on the context where they occur.
Now the statements from the body of SP are executed one by one that.
At the very beginning of the execution of a query the tables used in the
query are opened and open_and_process_table() now is called for each table
reference mentioned in the list of TABLE_LIST objects associated with the
query that was built when the query was parsed.
For each table reference first the reference is checked against CTEs
definitions in whose scope it occurred. If such definition is found the
reference is considered resolved and if this is not the first reference
to the found CTE the the specification of the CTE is re-parsed and the
result of the parsing is added to the parsing tree of the query as a
sub-tree. If this sub-tree contains table references to other tables they
are added to the list of TABLE_LIST objects associated with the query in
order the referenced tables to be opened. When the procedure that opens
the tables comes to the TABLE_LIST object created for a non-first
reference to a CTE it discovers that the referenced table instance is not
locked and reports an error.
Thus processing non-first table references to a CTE similar to how
references to view are processed does not work for queries used in stored
procedures / functions. And the main problem is that the current
pre-locking mechanism employed for stored procedures / functions does not
allow to save the context in which a CTE reference occur. It's not trivial
to save the info about the context where a CTE reference occurs while the
resolution of the table reference cannot be done without this context and
consequentially the specification for the table reference cannot be
determined.
This patch solves the above problem by moving resolution of all CTE
references at the parsing stage. More exactly references to CTEs occurred in
a query are resolved right after parsing of the query has finished. After
resolution any CTE reference it is marked as a reference to to derived
table. So it is excluded from the hash table created for pre-locking used
base tables and view when the first call of a stored procedure / function
is processed.
This solution required recursive calls of the parser. The function
THD::sql_parser() has been added specifically for recursive invocations of
the parser.
Item_in_subselect::create_single_in_to_exists_cond() should handle the
case where the subquery is a table-less select but it is not a result
of a UNION.
(Table-less subqueries like "(SELECT 1)" are "substituted" with their select
list, but table-less subqueries with WHERE or HAVING clause, like
"(SELECT 1 WHERE ...)" are not substituted. They are handled with regular
execution path)
m_status == DA_OK_BULK' failed in Diagnostics_area::message from
get_schema_tables_record
Analysis: SET NAMES changes character set for character_set_client,
character_set_connection, character_set_results to 'filename'. The .frm file of view
has @xx sequences in the SELECT query, which give parsing error because 'filename'
character set is not parser friendly. When we get parsing error (ER_PARSE_ERROR), we
directly return true without setting error status. This is caught later in assertion.
Fix: Disallow 'filename' character set in SET NAMES because it is not parser
friendly.
before change test:
strace -fe trace=file -o /tmp/f.strace sql/mysqld --datadir=/tmp/d --log-bin=foo-bin --help --verbose && ls -la /tmp/
...
'mysqladmin variables' instead of 'mysqld --verbose --help'.
total 0
drwxrwxr-x. 2 dan dan 60 May 19 18:05 .
drwxrwxrwt. 27 root root 640 May 19 18:03 ..
-rw-rw----. 1 dan dan 0 May 19 18:05 foo-bin.index
In Item_field::fix_fields(): when the item was resolved to an Item_field
in the SELECT's select_list, copy the Item_field's "depended_from" field.
Failure to do so caused the item to have incorrect attributes: it pointed
to a Field in an upper select but used_tables() didn't return
OUTER_REF_TABLE_BIT.
1) This commit implements reading all sections from configuration
files while looking for the current value of any server variable,
which were previously only read from the [mysqld.suffix] group and
from [mysqld], but not from other groups such as [mariadb.suffix],
[mariadb] or, for example, [server].
2) This commit also fixes misrecognition of some parameters when
parsing a command line containing a special marker for the end
of the list of options ("--") or when short option names (such
as "-s", "-a" and "-h arg") chained together (like a "-sah arg").
Such parameters can be passed to the SST script in the list of
arguments after "--mysqld-args" if the server is started with a
complex set of options - this was revealed during manual testing
of changes to read configuration files.
3) The server-side preparation code for the "--mysqld-args"
option list has also been simplified to make it easier to change
in the future (if needed), and has been improved to properly
handle the special backquote ("`") character in the argument
values.
Problem:
========
Aborting OPTIMIZE TABLE still logs in binary logs and replicates to the
Slave server. "Optimize table" command under execution, is killed by using
"Ctrl-C" as shown below.
MariaDB [test]> optimize table t2;
^CCtrl-C -- query killed. Continuing normally.
In spite of query execution being interrupted the query gets written to
binary log.
Analysis:
========
Admin command execution logic is not handling KILL command, hence it
ignores the KILL command and completes its execution.
Fix:
===
Check for thread killed notification, during admin command execution and
handle it. If thread kill occurs prior to any table modification the query
will not be written to binary log. If kill happens after at least one table
is modified then the query will be written to binary log. Ex: command in
execution is 'OPTIMIZE TABLE t1,t2' and the thread kill happens after t1
table is modified then 'OPTIMIZE TABLE t1,t2' will be written to binary log
as admin commands will not make the slave to diverge from master.
Problem:
=======
In slave_parallel_mode=optimistic configuration, when admin commands and
DML operation on the same table are scheduled simultaneously for execution,
it results in lock conflict and slave server either hangs due to
deadlock or goes down with an assert.
Analysis:
========
Admin commands OPTIMIZE, REPAIR and ANALYZE are written to binary log as
ordinary transactions. When 'slave_parallel_mode' is 'optimistic' DMLs are
allowed to run in parallel. But these locks are not detected by parallel
replication deadlock detection-and-handling mechanism. At times they result
in deadlock or assertion.
Fix:
===
Flag admin commands as DDL in Gtid_log_event at the time of writing to
binary log. Add a new bit EXECUTED_TABLE_ADMIN_CMD to
'm_unsafe_rollback_flags'. During 'mysql_admin_table' command execution it
accepts a list of tables to be processed and executes them in a loop. Upon
successful execution enable 'EXECUTED_TABLE_ADMIN_CMD' bit in
thd->transaction.stmt_unsafe_rollback_flags. Gtid_log_event constructor
will notice this flag and mark the current transaction with 'FL_DDL' flag.
Gtid_log_events marked as FL_DDL will not be scheduled parallel execution,
on the slave. They will execute in isolation to prevent deadlocks.
Note: Removed the call to 'trans_commit_implicit' from 'mysql_admin_table'
function as 'mysql_execute_command' will take care of invoking
'trans_commit_implicit'.
No longer a MySQL server, "his" is the wrong pronoun
for a server.
Thanks Michael Newton for highlighting these problems
Also changed slave -> replica.
1) This commit implements reading all sections from configuration
files while looking for the current value of any server variable,
which were previously only read from the [mysqld.suffix] group and
from [mysqld], but not from other groups such as [mariadb.suffix],
[mariadb] or, for example, [server].
2) This commit also fixes misrecognition of some parameters when
parsing a command line containing a special marker for the end
of the list of options ("--") or when short option names (such
as "-s", "-a" and "-h arg") chained together (like a "-sah arg").
Such parameters can be passed to the SST script in the list of
arguments after "--mysqld-args" if the server is started with a
complex set of options - this was revealed during manual testing
of changes to read configuration files.
3) The server-side preparation code for the "--mysqld-args"
option list has also been simplified to make it easier to change
in the future (if needed), and has been improved to properly
handle the special backquote ("`") character in the argument
values.
If a select query contained an ORDER BY clause that followed a LIMIT clause
or an ORDER BY clause or ORDER BY with LIMIT the EXPLAIN output for the
query showed an execution plan different from that was actually executed.
Approved by Roman Nozdrin <roman.nozdrin@mariadb.com>
Problem:- When we issue FTWRL with shutdown in parallel, there is race between
FTWRL and shutdown. Shutdown might destroy the mutex (pool->LOCK_rpl_thread_pool)
before FTWRL can lock it. So we can get crash on FTWRL thread
Solution:- mysql_mutex_destroy(pool->LOCK_rpl_thread_pool) should wait for
FTWRL thread to complete its work , and then destroy.
So slave_prepare_for_shutdown will just deactivate the pool, and mutex is destroyed
later in end_slave()
Parallel slave server shutdown found to be hanging in
close_connections() triggered by shutdown due to a slave worker thread
would not be notified to exit in case the worker was sitting idle.
Fixed with destroying the worker pool earlier that is in
slave_prepare_for_shutdown() when all their driver threads have already left.
A test file is added to simulate the bug condition as well as check
multi-sourced and not-idle worker cases.
XA transaction only allows to access data in specific states,
in ACTIVE, but not in IDLE or PREPARE.
But even then one should be able to run SHOW STATUS.
and configuration.
1. Pass joiner's authentication information to donor together with address
in State Transfer Request. This allows joiner to authenticate donor on
connection. Previously joiner would accept data from anywhere.
2. Deprecate custom SSL configuration variables tca, tcert and tkey in favor
of more familiar ssl-ca, ssl-cert and ssl-key. For backward compatibility
tca, tcert and tkey are still supported.
3. Allow falling back to server-wide SSL configuration in [mysqld] if no SSL
configuration is found in [sst] section of the config file.
4. Introduce ssl-mode variable in [sst] section that takes standard values
and has following effects:
- old-style SSL configuration present in [sst]: no effect
otherwise:
- ssl-mode=DISABLED or absent: retains old, backward compatible behavior
and ignores any other SSL configuration
- ssl-mode=VERIFY*: verify joiner's certificate and CN on donor,
verify donor's secret on joiner
(passed to donor via State Transfer Request)
BACKWARD INCOMPATIBLE BEHAVIOR
- anything else enables new SSL configuration convetions but does not
require verification
ssl-mode should be set to VERIFY only in a fully upgraded cluster.
Examples:
[mysqld]
ssl-cert=/path/to/cert
ssl-key=/path/to/key
ssl-ca=/path/to/ca
[sst]
-- server-wide SSL configuration is ignored, SST does not use SSL
[mysqld]
ssl-cert=/path/to/cert
ssl-key=/path/to/key
ssl-ca=/path/to/ca
[sst]
ssl-mode=REQUIRED
-- use server-wide SSL configuration for SST but don't attempt to
verify the peer identity
[sst]
ssl-cert=/path/to/cert
ssl-key=/path/to/key
ssl-ca=/path/to/ca
ssl-mode=VERIFY_CA
-- use SST-specific SSL configuration for SST and require verification
on both sides
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
When you only need view structure, don't call handle_derived with
DT_CREATE and rely on its internal hackish check to skip DT_CREATE.
Because handle_derived is called from many different places,
and this internal hackish check is indiscriminative.
Instead, just don't ask handle_derived to do DT_CREATE
if you don't want it to do DT_CREATE.
When you only need view structure, don't call handle_derived with
DT_CREATE and rely on its internal hackish check to skip DT_CREATE.
Because handle_derived is called from many different places,
and this internal hackish check is indiscriminative.
Instead, just don't ask handle_derived to do DT_CREATE
if you don't want it to do DT_CREATE.
The problem was caused by the following scenario:
Subquery's table has two indexes, KEY a(a), KEY a_b(a,b)
- LATERAL DERIVED optimization decides to use index a.
= The subquery uses ref access over key a.
- test_if_skip_sort_order() sees that KEY a_b satisfies the
subquery's GROUP BY clause, and attempts to switch to it.
= It fails to do so, because KEYUSE objects for index a_b
are switched off.
Fixed by disallowing to change the ref access key if it uses KEYUSE
objects injected by LATERAL DERIVED optimization.
