Long UNIQUE HASH index silently creates virtual column index, which should
be impossible for base columns featuring AUTO_INCREMENT.
Fix: add a relevant check; add new vcol type for a prettier error message.
Also fixes:
MDEV-25399 Assertion `name.length == strlen(name.str)' failed in Item_func_sp::make_send_field
Also fixes a problem that in this scenario:
SET NAMES binary;
SELECT 'some not well-formed utf8 string';
the auto-generated column name copied the binary string value directly
to the Item name, without checking utf8 well-formedness.
After this change auto-generated column names work as follows:
- Zero bytes 0x00 are copied to the name using HEX notation
- In case of "SET NAMES binary", all bytes sequences that do not make
well-formed utf8 characters are copied to the name using HEX notation.
Assertion `!pk->has_virtual()' failed in dict_index_build_internal_clust
while creating PRIMARY key longer than possible to store in the page.
This happened because the key was wrongly deduced as Long UNIQUE supported,
however PRIMARY KEY cannot be of that type. The main reason is that
only 8 bytes are used to store the hash, see HA_HASH_FIELD_LENGTH.
This is also why HA_NOSAME flag is removed (and caused the assertion in
turn) in open_table_from_share:
if (key_info->algorithm == HA_KEY_ALG_LONG_HASH)
{
key_part_end++;
key_info->flags&= ~HA_NOSAME;
}
To make it unique, the additional check is done by
check_duplicate_long_entries call from ha_write_row, and similar one from
ha_update_row.
PRIMARY key is already forbidden, which is checked by the first test in
main.long_unique, however is_hash_field_needed was wrongly deduced to true
in mysql_prepare_create_table in this particular case.
FIX:
* Improve the check for Key::PRIMARY type
* Simplify is_hash_field_needed deduction for a more neat reading
MySQL-5.7 mysql.user tables have a last_password_changed field.
Because before MariaDB-10.4 remained oblivious to this, the act of creating
users or otherwise changing a users row left the last_password_field with 0.
Running a MariaDB-10.4 instance on this would work correctly, until mysql_upgrade
is run, when this 0 value immediately translates to password expired
state.
MySQL-5.7 relied on the password_expired enum to indicate password
expiry so we aren't going to activate password that were expired in
MySQL-5.7.
Thanks Hans Borresen for the bug report and review of the fix.
Uninitialized ref_pointer_array[] because setup_fields() got empty
fields list. mysql_multi_update() for some reason does that by
substituting the fields list with empty total_list for the
mysql_select() call (looks like wrong merge since total_list is not
used anywhere else and is always empty). The fix would be to return
back the original fields list. But this fails update_use_source.test
case:
--error ER_BAD_FIELD_ERROR
update v1 set t1c1=2 order by 1;
Actually not failing the above seems to be ok.
The other fix would be to keep resolve_in_select_list false (and that
keeps outer context from being resolved in
Item_ref::fix_fields()). This fix is more consistent with how SELECT
behaves:
--error ER_SUBQUERY_NO_1_ROW
select a from t1 where a= (select 2 from t1 having (a = 3));
So this patch implements this fix.
There are two fill_record() functions (lines 8343 and 8618). First one
is used when there are some explicit values, the second one is used
for all implicit values. First one does update_default_fields(), the
second one did not. Added update_default_fields() call to the implicit
version of fill_record().
Changes on top of Sachin’s patch. Specifically:
1) Refined the parsing break condition to only change the parser’s
behavior for parsing strings in binary mode (behavior of \0 outside
of strings is unchanged).
2) Prefixed binary_zero_insert.test with ‘mysql_’ to more clearly
associate the purpose of the test.
3) As the input of the test contains binary zeros (0x5c00),
different text editors can visualize this sequence differently, and
Github would not display it at all. Therefore, the input itself was
consolidated into the test and created out of hex sequences to make
it easier to understand what is happening.
4) Extended test to validate that the rows which correspond to the
INSERTS with 0x5c00 have the correct binary zero data.
Reviewed By:
===========
Andrei Elkin <andrei.elkin@mariadb.com>
Problem:- Some binary data is inserted into the table using Jconnector. When
binlog dump of the data is applied using mysql cleint it gives syntax error.
Reason:-
After investigating it turns out to be a issue of mysql client not able to properly
handle \\\0 <0 in binary>. In all binary files where mysql client fails to insert
these 2 bytes are commom (0x5c00)
Solution:-
I have changed mysql.cc to include for the possibility that binary string can
have \\\0 in it
Normally we disable caching of routines in "SHOW CREATE".
Introduce an exception, if debug_dbug="+d,cache_sp_in_show_create".
lock_sync.test needs a way to populate the cache without side effects,
or else it runs into debug_sync timeouts.
So, this possibility to cache will be remain only for very special tests.
The reason for this behavior is that SP get cached, per connection.
The stored_program_cache is size of this cache, which amounts to 256
routines by default. A compiled stored procedure can easily be several
megabytes in size. Thus calling SHOW CREATE PROCEDURE for all stored
procedures, like mysqldump does, can require significant amount of memory.
Fixed by bypassing the cache for "SHOW CREATE". This should normally be
fine also perfomance-wise, as cache is meant to be used for repeated
execution, not repeated SHOW CREATEs.
Added a test to verify that CREATE PROCEDURE + SHOW CREATE PROCEURE do not
cache, i.e amount of allocated memory does not change.
Note, there is a change in existing behavior in an edge case :
If "SHOW CREATE PROCEDURE p1" called from p1, after p1 was altered, now
this will now return altered code. Previour behavior - relied on caching
and would return old code. The previous behavior might was not necessarily
correct.
instead of original name of the column
When doing refactoring of temporary table field creation a mistake was
done when copying the column name when creating internal temporary tables.
For internal temporary tables we should use the original field name, not
the item name (= alias).
Analysis: get_defaults_options() rewrites the value for my_defaults_file,
my_defaults_extra_file and my_defaults_group_suffix to 0. So the config file
can't be read.
Fix: Let handle_options() handle --defaults* option and create a new command
line to pass it to load_defaults().
This patch fixes parsing problems concerning derived tables that use table
value constructors (TVC) with LIMIT and ORDER BY clauses of the form
((VALUES ... LIMIT ...) ORDER BY ...) as dt
The fix has to be applied only to 10.3 as 10.4 that employs a different
grammar rules has no such problems. The test cases should be merged
upstream.
Approved by Oleksandr Byelkin <sanja@mariadb.com>
Setting specific ciphers does not work for OpenSSL 1.1.1+ client
when it negotiates TLSv1.3 with the server.
Require TLS version less that 1.3 for the server to workaround.
Analysis: When we have INSERT/REPLACE returning with qualified asterisk in the
RETURNING clause, '*' is not resolved properly because of wrong context.
context->table_list is NULL or has incorrect table because context->table_list
has tables from the FROM clause. For INSERT/REPLACE...SELECT...RETURNING,
context->table_list has table we are inserting from. While in other
INSERT/REPLACE syntax, context->table_list is NULL because there is no FROM
clause.
Fix: If filling fields instead of '*' for qualified asterisk in RETURNING,
use first_name_resolution_table for correct resolution of item.
In case a stored procedure is invoked in PS mode with argument of type ROW()
like the following one:
CALL p1(ROW(10,20))
such statement fails with the error
ER_OPERAND_COLUMNS (1241): Operand should contain 1 column(s)
The reason of emitting the error is that wrong method was invoked
on fixing an item corresponding to an argument of stored procedure -
the method fix_fields_if_needed_for_scalar() was called instead of
fix_fields_if_needed() that should be called.
This bug could affect queries that had at least two references to a CTE that
used an embedded recursive CTE.
Starting from version 10.4 some code in With_element::clone_parsed_spec()
that assumed a certain order of selects after parsing the specification of
a CTE became not valid anymore. It could lead to global select lists where
some selects were missing. If a missing CTE happened to belong to the
recursive part of a recursive CTE some recursive table references were not
set as references to materialized derived tables and this caused a crash of
the server.
Approved by Oleksandr Byelkin <sanja@mariadb.com>
Port the following patch from MySQL:
commit 1b2e8ea269c80cb93cc79d8be934c40b1c58e947
Author: Kailasnath Nagarkar <kailasnath.nagarkar@oracle.com>
Date: Fri Nov 30 16:43:13 2018 +0530
Bug #20939184: INNODB: UNLOCK ROW COULD NOT FIND A 2 MODE
LOCK ON THE RECORD
Issue:
------
Consdier tables t1 and t2 such that t1 has multiple rows
and join condition for t1 left join t2 results in only
single row from t2.
In this case, access to table t2 is const since there
is a single row that qualifies the join condition.
However, while executing the query, attempt is made to
unlock t2's row multiple times.
The current algorithm to fetch rows approximates to:
1) Retrieve the row for t1.
2) Retrieve the row for t2.
3) Apply the join conditions.
a) If condition evaluates to true:
Project the row to the result.
b) If condition evaluates to false:
i) If t2's qep_tab->not_null_complement is true,
unlock t2's row.
ii) Null-complement the row by calling
"evaluate_null_complemented_join_record()". In
this function qep_tab->not_null_complement is
set to false.
The t2's only one row, that qualifies join condition,
is unlocked in Step i) when t1's row is evaluated to
false.
When t1's next row is also evaluated to false, another
attempt is made to unlock t2's already unlocked row.
This results in following error being logged in error.log:
"[ERROR] InnoDB: Unlock row could not find a 3 mode lock on
the record. Current statement:
select * from t1 left join t2 ......"
Solution:
---------
When a table's access method is "const", set record unlock
method for this table to do no operation.
This bug affected queries with views / derived_tables / CTEs whose
specifications were of the form
(SELECT ... LIMIT <n>) ORDER BY ...
Units representing such specifications contains one SELECT_LEX structure
for (SELECT ... LIMIT <n>) and additionally SELECT_LEX structure for
fake_select_lex. This fact should have been taken into account in the
function mysql_derived_fill().
This patch has to be applied to 10.2 and 10.3 only.
Add fixed for tests mysqld--help,aix.rdiff and sysvars_server_notembedded,aix.rdiff
AIX couldn't compile in embedded mode so leaving sysvars_server_embedded
for later (if required).
The check-testcase record uses a mysqltest connection
to the database to do the recording. With the server configured
as an abstract socket, the mysqltest client cannot connect and
fails.
We work around this by starting the server as normal and then
restart with an abstract socket and test this.
This didn't affect Windows as it just did a tcp connection.
So this did affect all unix socket based systems except Linux
as this was the only one that supported abstract sockets.
the idea of main.failed_auth_unixsocket was to have existing
user account (root) authenticate with unix_socket, then login with
non-existent user name, Non-existent user name forces the server
to perform the authentication in the name of some random existing
user. But it must still fail at the end, as the user name is wrong.
In 10.4 a second predefined user was added, mariadb.sys, so root
is not the only user in mysql.global_priv and unix_socket auth
must be forced for all existing user accounts, because we cannot
know what user account the server will randomly pick for non-existing
user auth.
If a select query contained an ORDER BY clause that followed a LIMIT clause
or an ORDER BY clause or ORDER BY with LIMIT the EXPLAIN output for the
query showed an execution plan different from that was actually executed.
Approved by Roman Nozdrin <roman.nozdrin@mariadb.com>
If a join query uses a derived table (view / CTE) with GROUP BY clause then
the execution plan for such join may employ split optimization. When this
optimization is employed the derived table is not materialized. Rather only
some partitions of the derived table are subject to grouping. Split
optimization can be applied only if:
- there are some indexes over the tables used in the join specifying the
derived table whose prefixes partially cover the field items used in the
GROUP BY list (such indexes are called splitting indexes)
- the WHERE condition of the join query contains conjunctive equalities
between columns of the derived table that comprise major parts of
splitting indexes and columns of the other join tables.
When the optimizer evaluates extending of a partial join by the rows of the
derived table it always considers a possibility of using split optimization.
Different splitting indexes can be used depending on the extended partial
join. At some rare conditions, for example, when there is a non-splitting
covering index for a table joined in the join specifying the derived table
usage of a splitting index to produce rows needed for grouping may be still
less beneficial than usage of such covering index without any splitting
technique. The function JOIN_TAB::choose_best_splitting() must take this
into account.
Approved by Oleksandr Byelkin <sanja@mariadb.com>
If a join query uses a derived table (view / CTE) with GROUP BY clause then
the execution plan for such join may employ split optimization. When this
optimization is employed the derived table is not materialized. Rather only
some partitions of the derived table are subject to grouping. Split
optimization can be applied only if:
- there are some indexes over the tables used in the join specifying the
derived table whose prefixes partially cover the field items used in the
GROUP BY list (such indexes are called splitting indexes)
- the WHERE condition of the join query contains conjunctive equalities
between columns of the derived table that comprise major parts of
splitting indexes and columns of the other join tables.
When the optimizer evaluates extending of a partial join by the rows of the
derived table it always considers a possibility of using split optimization.
Different splitting indexes can be used depending on the extended partial
join. At some rare conditions, for example, when there is a non-splitting
covering index for a table joined in the join specifying the derived table
usage of a splitting index to produce rows needed for grouping may be still
less beneficial than usage of such covering index without any splitting
technique. The function JOIN_TAB::choose_best_splitting() must take this
into account.
Approved by Oleksandr Byelkin <sanja@mariadb.com>
The problem was that when LOCK TABLES where unwinded as part of
a killed connection, unlink_all_closed_tables() did not like that
there was uncommited transactions.
Fixed by doing a rollback of any open transaction in this particular case.
In the code existed just before this patch binding of a table reference to
the specification of the corresponding CTE happens in the function
open_and_process_table(). If the table reference is not the first in the
query the specification is cloned in the same way as the specification of
a view is cloned for any reference of the view. This works fine for
standalone queries, but does not work for stored procedures / functions
for the following reason.
When the first call of a stored procedure/ function SP is processed the
body of SP is parsed. When a query of SP is parsed the info on each
encountered table reference is put into a TABLE_LIST object linked into
a global chain associated with the query. When parsing of the query is
finished the basic info on the table references from this chain except
table references to derived tables and information schema tables is put
in one hash table associated with SP. When parsing of the body of SP is
finished this hash table is used to construct TABLE_LIST objects for all
table references mentioned in SP and link them into the list of such
objects passed to a pre-locking process that calls open_and_process_table()
for each table from the list.
When a TABLE_LIST for a view is encountered the view is opened and its
specification is parsed. For any table reference occurred in
the specification a new TABLE_LIST object is created to be included into
the list for pre-locking. After all objects in the pre-locking have been
looked through the tables mentioned in the list are locked. Note that the
objects referenced CTEs are just skipped here as it is impossible to
resolve these references without any info on the context where they occur.
Now the statements from the body of SP are executed one by one that.
At the very beginning of the execution of a query the tables used in the
query are opened and open_and_process_table() now is called for each table
reference mentioned in the list of TABLE_LIST objects associated with the
query that was built when the query was parsed.
For each table reference first the reference is checked against CTEs
definitions in whose scope it occurred. If such definition is found the
reference is considered resolved and if this is not the first reference
to the found CTE the the specification of the CTE is re-parsed and the
result of the parsing is added to the parsing tree of the query as a
sub-tree. If this sub-tree contains table references to other tables they
are added to the list of TABLE_LIST objects associated with the query in
order the referenced tables to be opened. When the procedure that opens
the tables comes to the TABLE_LIST object created for a non-first
reference to a CTE it discovers that the referenced table instance is not
locked and reports an error.
Thus processing non-first table references to a CTE similar to how
references to view are processed does not work for queries used in stored
procedures / functions. And the main problem is that the current
pre-locking mechanism employed for stored procedures / functions does not
allow to save the context in which a CTE reference occur. It's not trivial
to save the info about the context where a CTE reference occurs while the
resolution of the table reference cannot be done without this context and
consequentially the specification for the table reference cannot be
determined.
This patch solves the above problem by moving resolution of all CTE
references at the parsing stage. More exactly references to CTEs occurred in
a query are resolved right after parsing of the query has finished. After
resolution any CTE reference it is marked as a reference to to derived
table. So it is excluded from the hash table created for pre-locking used
base tables and view when the first call of a stored procedure / function
is processed.
This solution required recursive calls of the parser. The function
THD::sql_parser() has been added specifically for recursive invocations of
the parser.
# Conflicts:
# sql/sql_cte.cc
# sql/sql_cte.h
# sql/sql_lex.cc
# sql/sql_lex.h
# sql/sql_view.cc
# sql/sql_yacc.yy
# sql/sql_yacc_ora.yy
In the code existed just before this patch binding of a table reference to
the specification of the corresponding CTE happens in the function
open_and_process_table(). If the table reference is not the first in the
query the specification is cloned in the same way as the specification of
a view is cloned for any reference of the view. This works fine for
standalone queries, but does not work for stored procedures / functions
for the following reason.
When the first call of a stored procedure/ function SP is processed the
body of SP is parsed. When a query of SP is parsed the info on each
encountered table reference is put into a TABLE_LIST object linked into
a global chain associated with the query. When parsing of the query is
finished the basic info on the table references from this chain except
table references to derived tables and information schema tables is put
in one hash table associated with SP. When parsing of the body of SP is
finished this hash table is used to construct TABLE_LIST objects for all
table references mentioned in SP and link them into the list of such
objects passed to a pre-locking process that calls open_and_process_table()
for each table from the list.
When a TABLE_LIST for a view is encountered the view is opened and its
specification is parsed. For any table reference occurred in
the specification a new TABLE_LIST object is created to be included into
the list for pre-locking. After all objects in the pre-locking have been
looked through the tables mentioned in the list are locked. Note that the
objects referenced CTEs are just skipped here as it is impossible to
resolve these references without any info on the context where they occur.
Now the statements from the body of SP are executed one by one that.
At the very beginning of the execution of a query the tables used in the
query are opened and open_and_process_table() now is called for each table
reference mentioned in the list of TABLE_LIST objects associated with the
query that was built when the query was parsed.
For each table reference first the reference is checked against CTEs
definitions in whose scope it occurred. If such definition is found the
reference is considered resolved and if this is not the first reference
to the found CTE the the specification of the CTE is re-parsed and the
result of the parsing is added to the parsing tree of the query as a
sub-tree. If this sub-tree contains table references to other tables they
are added to the list of TABLE_LIST objects associated with the query in
order the referenced tables to be opened. When the procedure that opens
the tables comes to the TABLE_LIST object created for a non-first
reference to a CTE it discovers that the referenced table instance is not
locked and reports an error.
Thus processing non-first table references to a CTE similar to how
references to view are processed does not work for queries used in stored
procedures / functions. And the main problem is that the current
pre-locking mechanism employed for stored procedures / functions does not
allow to save the context in which a CTE reference occur. It's not trivial
to save the info about the context where a CTE reference occurs while the
resolution of the table reference cannot be done without this context and
consequentially the specification for the table reference cannot be
determined.
This patch solves the above problem by moving resolution of all CTE
references at the parsing stage. More exactly references to CTEs occurred in
a query are resolved right after parsing of the query has finished. After
resolution any CTE reference it is marked as a reference to to derived
table. So it is excluded from the hash table created for pre-locking used
base tables and view when the first call of a stored procedure / function
is processed.
This solution required recursive calls of the parser. The function
THD::sql_parser() has been added specifically for recursive invocations of
the parser.
Problem:
=======
The ALWAYS option of the mariadb-binlog --base64-output flag
formats its output incorrectly. This option is deprecated, and
MySQL 8.0 has removed it entirely.
Solution:
========
Adhere to MySQL and remove this option from MariaDB.
Behavioral Changes:
==================
Use Case: ./mariadb-binlog --base64-output
Previous Behavior: Sets base64-output mode to always
New Behavior: Error message indicating incomplete argument
Use Case: ./mariadb-binlog --base64-output=always
Previous Behavior: Sets base64-output mode to always
New Behavior: Error message indicating invalid argument value
Reviewed By:
==========
Andrei Elkin: <andrei.elkin@mariadb.com>
The problem was caused by the following scenario:
Subquery's table has two indexes, KEY a(a), KEY a_b(a,b)
- LATERAL DERIVED optimization decides to use index a.
= The subquery uses ref access over key a.
- test_if_skip_sort_order() sees that KEY a_b satisfies the
subquery's GROUP BY clause, and attempts to switch to it.
= It fails to do so, because KEYUSE objects for index a_b
are switched off.
Fixed by disallowing to change the ref access key if it uses KEYUSE
objects injected by LATERAL DERIVED optimization.
remove code duplication in Lex_input_stream::scan_ident_middle(),
make sure identifiers are always use the same code path whether
they start form an underscore or not.
The optimizer removes redundant GROUP BY operations. If GROUP BY element
is a subselect, it is "eliminated".
However one must not eliminate the item if it is used both in the select
list and in the GROUP BY, like so:
select (select ... ) as SUBQ from ... group by SUBQ
Do not eliminate such items.
Before this patch mergeable derived tables / view used in a multi-table
update / delete were merged before the preparation stage.
When the merge of a derived table / view is performed the on expression
attached to it is fixed and ANDed with the where condition of the select S
containing this derived table / view. It happens after the specification of
the derived table / view has been merged into S. If the ON expression refers
to a non existing field an error is reported and some other mergeable derived
tables / views remain unmerged. It's not a problem if the multi-table
update / delete statement is standalone. Yet if it is used in a stored
procedure the select with incompletely merged derived tables / views may
cause a problem for the second call of the procedure. This does not happen
for select queries using derived tables / views, because in this case their
specifications are merged after the preparation stage at which all ON
expressions are fixed.
This patch makes sure that merging of the derived tables / views used in a
multi-table update / delete statement is performed after the preparation
stage.
Approved by Oleksandr Byelkin <sanja@mariadb.com>
Issue:- Since there is no waiting for the actual disconnection of the con_tmp
(which does XA prepare of test1), We can have a issue when test1 is not
prepared and we are calling rollback on test1 , giving XAER_NOTA: Unknown XID
error
Solution:- Wait for the complete disconnection of con_tmp
The easiest way to compile and test the server with UBSAN is to run:
./BUILD/compile-pentium64-ubsan
and then run mysql-test-run.
After this commit, one should be able to run this without any UBSAN
warnings. There is still a few compiler warnings that should be fixed
at some point, but these do not expose any real bugs.
The 'special' cases where we disable, suppress or circumvent UBSAN are:
- ref10 source (as here we intentionally do some shifts that UBSAN
complains about.
- x86 version of optimized int#korr() methods. UBSAN do not like unaligned
memory access of integers. Fixed by using byte_order_generic.h when
compiling with UBSAN
- We use smaller thread stack with ASAN and UBSAN, which forced me to
disable a few tests that prints the thread stack size.
- Verifying class types does not work for shared libraries. I added
suppression in mysql-test-run.pl for this case.
- Added '#ifdef WITH_UBSAN' when using integer arithmetic where it is
safe to have overflows (two cases, in item_func.cc).
Things fixed:
- Don't left shift signed values
(byte_order_generic.h, mysqltest.c, item_sum.cc and many more)
- Don't assign not non existing values to enum variables.
- Ensure that bool and enum values are properly initialized in
constructors. This was needed as UBSAN checks that these types has
correct values when one copies an object.
(gcalc_tools.h, ha_partition.cc, item_sum.cc, partition_element.h ...)
- Ensure we do not called handler functions on unallocated objects or
deleted objects.
(events.cc, sql_acl.cc).
- Fixed bugs in Item_sp::Item_sp() where we did not call constructor
on Query_arena object.
- Fixed several cast of objects to an incompatible class!
(Item.cc, Item_buff.cc, item_timefunc.cc, opt_subselect.cc, sql_acl.cc,
sql_select.cc ...)
- Ensure we do not do integer arithmetic that causes over or underflows.
This includes also ++ and -- of integers.
(Item_func.cc, Item_strfunc.cc, item_timefunc.cc, sql_base.cc ...)
- Added JSON_VALUE_UNITIALIZED to json_value_types and ensure that
value_type is initialized to this instead of to -1, which is not a valid
enum value for json_value_types.
- Ensure we do not call memcpy() when second argument could be null.
- Fixed that Item_func_str::make_empty_result() creates an empty string
instead of a null string (safer as it ensures we do not do arithmetic
on null strings).
Other things:
- Changed struct st_position to an OBJECT and added an initialization
function to it to ensure that we do not copy or use uninitialized
members. The change to a class was also motived that we used "struct
st_position" and POSITION randomly trough the code which was
confusing.
- Notably big rewrite in sql_acl.cc to avoid using deleted objects.
- Changed in sql_partition to use '^' instead of '-'. This is safe as
the operator is either 0 or 0x8000000000000000ULL.
- Added check for select_nr < INT_MAX in JOIN::build_explain() to
avoid bug when get_select() could return NULL.
- Reordered elements in POSITION for better alignment.
- Changed sql_test.cc::print_plan() to use pointers instead of objects.
- Fixed bug in find_set() where could could execute '1 << -1'.
- Added variable have_sanitizer, used by mtr. (This variable was before
only in 10.5 and up). It can now have one of two values:
ASAN or UBSAN.
- Moved ~Archive_share() from ha_archive.cc to ha_archive.h and marked
it virtual. This was an effort to get UBSAN to work with loaded storage
engines. I kept the change as the new place is better.
- Added in CONNECT engine COLBLK::SetName(), to get around a wrong cast
in tabutil.cpp.
- Added HAVE_REPLICATION around usage of rgi_slave, to get embedded
server to compile with UBSAN. (Patch from Marko).
- Added #ifdef for powerpc64 to avoid a bug in old gcc versions related
to integer arithmetic.
Changes that should not be needed but had to be done to suppress warnings
from UBSAN:
- Added static_cast<<uint16_t>> around shift to get rid of a LOT of
compiler warnings when using UBSAN.
- Had to change some '/' of 2 base integers to shift to get rid of
some compile time warnings.
Reviewed by:
- Json changes: Alexey Botchkov
- Charset changes in ctype-uca.c: Alexander Barkov
- InnoDB changes & Embedded server: Marko Mäkelä
- sql_acl.cc changes: Vicențiu Ciorbaru
- build_explain() changes: Sergey Petrunia
A user connected to a server with an expired password
can't change password with the statement "SET password=..."
if this statement is run in PS mode. In mentioned use case a user
gets the error ER_MUST_CHANGE_PASSWORD on attempt to run
the statement PREPARE stmt FOR "SET password=...";
The reason of failure to reset password by a locked user using the
statement PREPARE stmt FOR "SET password=..." is that PS-related
statements are not listed among the commands allowed for execution
by a user with expired password. However, simple adding of PS-related
statements (PREPARE FOR/EXECUTE/DEALLOCATE PREPARE ) to the list of
statements allowed for execution by a locked user is not enough
to solve problems, since it opens the opportunity for a locked user
to execute any statement in the PS mode.
To exclude this opportunity, additional checking that the statement
being prepared for execution in PS-mode is the SET statement has to be added.
This extra checking has been added by this patch into the method
Prepared_statement::prepared() that executed on preparing any statement
for execution in PS-mode.
Handle "col<>const" in the same way that MDEV-21958 did for
"col NOT IN(const-list)": do not use the condition for range/index_merge
accesses if there is a unique UNIQUE KEY(col).
The testcase is in main/range.test. The rest of test updates are
due to widespread use of 'pk<>1' in the testsuite. Changed the test
to use different but equivalent forms of the conditions.
Fixed by adding a MDL_BACKUP_COMMIT lock before altering temporary tables
whose creation was logged to binary log (in which case the ALTER TABLE
must also be logged)
splittable derived
If one of joined tables of the processed query is a materialized derived
table (or view or CTE) with GROUP BY clause then under some conditions it
can be subject to split optimization. With this optimization new equalities
are injected into the WHERE condition of the SELECT that specifies this
derived table. The injected equalities are generated for all join orders
with which the split optimization can employed. After the best join order
has been chosen only certain of this equalities are really needed. The
others can be safely removed. If it's not done and some of injected
equalities involve expressions over semi-joins with look-up access then
the query may return a wrong result set.
This patch effectively removes equalities injected for split optimization
that are needed only at the optimization stage and not needed for execution.
Approved by serg@mariadb.com
in ON condition
The fix of the bug MDEV-25002 for 10.4 turned out to be incomplete. It
caused crashes when executing CREATE VIEW, CREATE TABLE .. SELECT,
INSERT .. SELECT statements if their SELECTs contained references to
non-existing fields.
This patch complements the fix for MDEV-25002 in order to avoid such
crashes.
Approved by Oleksandr Byelkin <sanja@mariadb.com>
Note they key_or() may call tree_delete(), which will cause the weight
asserts to be checked. In order to avoid them from firing, update key1
tree's weight after we've changed key1->some_local_child->next_key_part.
Having done that, do we still need this at the function end:
/* Re-compute the result tree's weight. */
key1->update_weight_locally();
?
It was possibile for a user to create an interlocked state which may go on
for a significant period of time. There is a tight loop in the FTWRL code
path that tries to repeatedly acquire a read lock. As the weight of FTWRL
lock is the smallest among others, it's always selected by the deadlock
detector, but can never be killed.
Imaging the following sequence:
connection_0 connection_1
GET_LOCK("l1", 0);
LOCK TABLES t WRITE;
FLUSH TABLES WITH READ LOCK;
GET_LOCK("l1", 1000);
The GET_LOCK statement in connection_1 triggers the deadlock detector,
which tries to select the lock in FTWRL, since its weight is 0. However,
since a loop in Global_read_lock::lock_global_read_lock() tries to always
win, it tries to acquire lock again. Which invokes the deadlock detector,
and that cycle continues until GET_LOCK in connection_1 times out.
This patch resolves the live-locking by introducing a dynamic bonus to the
deadlock weight associated with every lock. Each lock gets a bonus weight
each time it's selected by the deadlock detector. In case of a live-lock
situation, those locks that cannot be killed, get additional weight each
iteration. Eventually their weight becomes so high that the deadlock
detector shifts its attention to other lock, until it find the one that
can be killed.
optimize_schema_tables_memory_usage() crashed when its argument included
TABLE struct that was not fully initialized.
To prevent such a crash, we check if a table is an information schema table at
the beginning of each iteration.
Closes#1768
not be dropped if the DEFINER is custom. Revert changes
to MDEV-23102 tests as they were designed to catch
this corner case.
The explanation for this corner case is that users
historically used to tweak the mysql.user table and
probably still do even though mysql.user is now a view.
Thus, if the DEFINER of the view is not default, i.e.
root@localhost or mariadb.sys@localhost, we should avoid
dropping the view during upgrade process to not discard
potential custom changes.
if a query used no fields from an I_S table, we were creating a temp
table with one, first, field (as a table cannot have zero fields),
with its length truncated to 1.
Now - force also this dummy field to be a normal field, not a BLOB
The mysql.user view password_expired column should display the right
result, in sync with whether an account has its password expired or not
For mariadb 10.4+ upgrades before this commit, the mysql.user view needs
to be dropped and recreated to actually make the view display the
correct value for the password_expired column.
The bug caused crashes of the server when processing queries with nested
table value constructors (TVC) . It happened because the grammar rules to
parse TVC used the same global lists for both nested TVC and nesting TVC.
As a result invalid select trees were constructed for queries with nested
TVC and this led to crashes at the prepare stage.
This patch provides its own lists structures for each TVC nest level.
Besides the patch fixes a bug in the function wrap_tvc() that missed
inheritance of the SELECT_LEX::exclude_from_table_unique_test for
selects that wrapped TVCs. This inheritance is critical for specifications
of derived tables that employ nested TVCs.
Approved by dmitry.shulga@mariadb.com
used in set function
If a subselect is formed by a table value constructor (TVC) then the
following transformation is applied at the prepare stage:
VALUES (v1), ... (vn) => SELECT * FROM (VALUES (v1), ... (vn)) tvc_x.
The transformation is performed by the function wrap_tvc() that resets
THD::LEX::current select to the top level select of the result of the
transformation. After the call of wrap_tvc() in the function
Item_subselect::wrap_tvc_into_select() the field THD::LEX::current must be
reset to the same select as before the call. It was not done. As a result
if the subselect formed by a TVC was an argument of a set function then
an assertion was hit in the function Item_sum::check_sum_func().
Approved by Oleksandr Byelkin <sanja@mariadb.com>
Fixes also:
MDEV-24942 Server crashes in _ma_rec_pack... with DEFAULT() on BLOB
This was caused by two different bugs, both related to that the default
value for the blob was not calculated before it was used:
- There where now Item_default_value::..result() wrappers, which is
needed as item in HAVING uses these. This causes crashes when
using a reference to a DEFAULT(blob_field) in HAVING. It also
caused wrong results when used with other fields with default value
expressions that are not constants.
- create_tmp_field() did not take into account that blob fields with
default expressions are not yet initialized. Fixed by treating
Item_default_value(blob) like a normal item expression.
The failure happened for group by queries when all tables where marked as
'const tables' (tables with 0-1 matching rows) and no row matched the
where clause and there was in addition a direct reference to a field.
In this case the field would not be properly reset and the query would
return 'random data' that happended to be in table->record[0].
Fixed by marking all const tables as null tables in this particular case.
Sergei also provided an extra test case for the code.
@reviewer Sergei Petrunia <psergey@askmonty.org>
This bug caused crashes of the server when processing queries with table
value constructors (TVC) that contained subqueries and were used itself as
subselects. For such TVCs the following transformation is applied at the
prepare stage:
VALUES (v1), ... (vn) => SELECT * FROM (VALUES (v1), ... (vn)) tvc_x.
This transformation allows to reduce the problem of evaluation of TVCs used
as subselects to the problem of evaluation of regular subselects.
The transformation is implemented in the wrap_tvc(). The code the function
to mimic the behaviour of the parser when processing the result of the
transformation. However this imitation was not free of some flaws. First
the function called the method exclude() that completely destroyed the
select tree structures below the transformed TVC. Second the function
used the procedure mysql_new_select to create st_select_lex nodes for
both wrapping select of the transformation and TVC. This also led to
constructing of invalid select tree structures.
The patch actually re-engineers the code of wrap_tvc().
Approved by Oleksandr Byelkin <sanja@mariadb.com>
The problem was in and_all_keys(), the code of MDEV-9759 which calculates
the new tree weight:
First, it didn't take into account the case when
(next->next_key_part=tmp) == NULL
and dereferenced a NULL pointer when getting tmp->weight.
Second, "if (param->alloced_sel_args > SEL_ARG::MAX_SEL_ARGS) break"
could leave the loop with incorrect value of weight.
Fixed by introducing SEL_ARG::update_weight_locally() and calling it
at the end of the function. This allows to avoid caring about all the
above cases.
Given PASSWORD EXPIRE and PASSWORD EXPIRE [NEVER|INTERVAL x DAY] are
two different mechanisms, SHOW CREATE USER should display all the
information required to restore the state of an account which
includes both a manual expired state and an automatic policy.
The solution proposed here keeps a CREATE USER ... PASSWORD EXPIRE
statement and adds an aditional
ALTER USER .. PASSWORD EXPIRE [NEVER|INTERVAL x DAY] when necessary
This way a tool can restore almost the complete state of an account
as it was before a dump. The only information left still is the
value of the password_last_changed column from mysql.global_priv
This was caused by two different bugs:
1) Information_schema tables where not locked by lock_tables, but
get_lock_data() was not filtering these out. This caused a crash when
mysql_unlock_some_tables() tried to unlock tables early, including
not locked information schema tables.
Fixed by not locking SYSTEM_TMP_TABLES
2) In some cases the optimizer will notice that we do not need to read
the information_schema tables at all. In this case
join_tab->read_record is not set, which caused a crash in
get_schema_tables_result()
Fixed by ignoring const tables in get_schema_tables_result()
on used subqueries
If a query was based on a table value constructor that contained subqueries
then EXPLAIN for such query did not contain any lines explaining the
execution plans of the subqueries.
This happened because
- no optimize() method was called for any subquery used by the table value
constructor when EXPLAIN command for the query was processed;
- EXPLAIN node created for the table value constructor itself did not
assume that some child nodes could be attached to it.
Approved by Oleksandr Byelkin <sanja@mariadb.com>
This can cause crashes when accessing already released memory
The issue was the Item_default created a internal field, pointing to
share->default_values, to be used with the DEFAULT() function.
This does not work for BLOB fields as these are freed at end of query.
Fixed by storing BLOB field data inside and area allocated by
Item_default_value, like we do for nondeterministic default values.
When creating a summary temporary table with bit fields used in the sum
expression with several parameters, like GROUP_CONCAT(), the counting of
bits needed in the record was wrong.
The reason we got an assert in Aria was because the bug caused a memory
overwrite in the record and Aria noticed that the data was 'impossible.
2 different problems:
- MYSQL_BIN_LOG::write() did not check if mdl_context.acquire_lock() failed
- Sql_cmd_optimize_table::execute() and Sql_cmd_repair_table::execute()
called write_bin_log(), which could fail if sql_admin() had already
called my_eof()
Fixed by adding check for aquire_lock() return status and protect
write_bin_log() in the above two functions with set_overwrite_status().
of two table value costructors
This bug affected queries with a [NOT] IN/ANY/ALL subquery whose top level
unit contained several table value constructors.
The problem appeared because the code of the function
Item_subselect::fix_fields() that was responsible for wrapping table
value constructors encountered at the top level unit of a [NOT] IN/ANY/ALL
subquery did not take into account that the chain of the select objects
comprising the unit were not immutable.
Approved by Oleksandr Byelkin <sanja@mariadb.com>
One should not change the program arguments!
This change also reduces warnings from the icc compiler.
Almost all changes are just syntax changes (adding const to
'get_one_option function' declarations).
Other changes:
- Added a few cast of 'argument' from 'const char*' to 'char *'. This
was mainly in calls to 'external' functions we don't have control of.
- Ensure that all reset of 'password command line argument' are similar.
(In almost all cases it was just adding a comment and a cast)
- In mysqlbinlog.cc and mysqld.cc there was a few cases that changed
the command line argument. These places where changed to instead allocate
the option in a MEM_ROOT to avoid changing the argument. Some of this
code was changed to ensure that different programs did parsing the
same way. Added a test case for the changes in mysqlbinlog.cc
- Changed a few variables that took their value from command line options
from 'char *' to 'const char *'.
partially revert 76063c2a13. Item::clone() is not an all-purpose
Item copying machine, it was specifically created for pushdown
of predicates into derived tables and views and it does not
copy everything. In particular, it does not copy Item_func_regex.
Fix the bug differently by preserving the old constraint name.
But keep setting automatic_name=true to have it regenerated
for cases like ALTER TABLE ... ADD CONSTRAINT.
Also update the SEL_ARG graph weight in:
- sel_add()
- SEL_ARG::clone()
Make key_{and,or}_with_limit() to also verify weight for the arguments
(There is no single point to verify SEL_ARG graphs constructed from
conditions that are not AND-OR formulas, so we hope that those are
connected with AND/OR and do it here).
(Variant #5, full patch, for 10.5)
Do not produce SEL_ARG graphs that would yield huge numbers of ranges.
Introduce a concept of SEL_ARG graph's "weight". If we are about to
produce a graph whose "weight" exceeds the limit, remove the parts
of SEL_ARG graph that represent the biggest key parts. Do so until
the graph's is within the limit.
Includes
- debug code to verify SEL_ARG graph weight
- A user-visible @@optimizer_max_sel_arg_weight to control the optimization
- Logging the optimization into the optimizer trace.
a. The change makes `mariadb-upgrade` detect if `MYSQL_JSON` data type is needed.
b. Install the data type if it's not installed.
c. Uninstalls the data type once finished.
d. Create `.opt` and `.inc` files `have_type_mysql_json` and adapt the
tests
Reviewed by: vicentiu@mariadb.org
This patch actually fixes the bug MDEV-24675 and the bug MDEV-24618:
Assertion failure when TVC uses a row in the context expecting scalar value
The cause of these bugs is the same wrong call of the function that fixes
value expressions in the value list of a table value constructor.
The assertion failure happened when an expression in the value list is of
the row type. In this case an error message was expected, but it was not
issued because the function fix_fields_if_needed() was called for to
check fields of value expressions in a TVC instead of the function
fix_fields_if_needed_for_scalar() that would also check that the value
expressions are are of a scalar type.
The first bug happened when a table value expression used an expression
returned by single-row subselect. In this case the call of the
fix_fields_if_needed_for_scalar virtual function must be provided with
and address to which the single-row subselect has to be attached.
Test cases were added for each of the bugs.
Approved by Oleksandr Byelkin <sanja@mariadb.com>
I_S tables were materialized too late, an attempt to use table
statistics before the table was created caused a crash.
Let's move table creation up. it only needs read_set to
be calculated properly, this happens in JOIN::optimize_inner(),
after semijoin transformation.
Note that tables are not populated at that point, so most of the
statistics would make no sense anyway. But at least field sizes
will be correct. And it won't crash.
On parsing statements for which a starting backtick (`) delimiter doesn't have
a corresponding ending backtick, a current pointer to a position inside a
pre-processed buffer could go beyond the end of the buffer.
This bug report caused by the commit d496765903
"MDEV-22022 Various mangled SQL statements will crash 10.3 to 10.5 debug builds".
In order to fix the issue both pointers m_ptr and m_cpp_ptr must be
rolled back to previous position in raw input and pre-processed input streams
correspondingly in case end of query reached during parsing.
constellation
Analysis: The decimals is set to NOT_FIXED_DEC for Field_str even if it is
NULL. Unsigned has decimals=0. So Type_std_attributes::decimals is set to 39
(maximum between 0 and 39). This results in incorrect number of decimals
when we have union of unsigned and NULL type.
Fix: Check if the field is created from NULL value. If yes, set decimals to 0
otherwise set it to NOT_FIXED_DEC.
the test was doing
--replace_result $con_id con_id
eval SHOW EXPLAIN FOR $con_id;
with the intention of replacing the variable part of the statement
in the result log. But actually replace_result replaces everything
that matches. In particular, when $con_id is 100, the warning
Note 1003 select sleep(100000)
becomes
Note con_id3 select sleep(con_id000)
don't expect return type of a stored function to be valid.
it's read from a table, so can be messed with.
it even can contain \0 bytes in the middle of the type name
