Fix the issue introduced in ec2574fd8f, fix for MDEV-31983:
get_quick_record_count() must set quick_count=0 when it got
IMPOSSIBLE_RANGE from test_quick_select.
Failure to do so will cause an assertion in 11.0, when the number of
quick select rows (0) is checked to be lower than the number of
found_records (which is capped up to 1).
The crash happened with an indexed virtual column whose
value is evaluated using a function that has a different meaning
in sql_mode='' vs sql_mode=ORACLE:
- DECODE()
- LTRIM()
- RTRIM()
- LPAD()
- RPAD()
- REPLACE()
- SUBSTR()
For example:
CREATE TABLE t1 (
b VARCHAR(1),
g CHAR(1) GENERATED ALWAYS AS (SUBSTR(b,0,0)) VIRTUAL,
KEY g(g)
);
So far we had replacement XXX_ORACLE() functions for all mentioned function,
e.g. SUBSTR_ORACLE() for SUBSTR(). So it was possible to correctly re-parse
SUBSTR_ORACLE() even in sql_mode=''.
But it was not possible to re-parse the MariaDB version of SUBSTR()
after switching to sql_mode=ORACLE. It was erroneously mis-interpreted
as SUBSTR_ORACLE().
As a result, this combination worked fine:
SET sql_mode=ORACLE;
CREATE TABLE t1 ... g CHAR(1) GENERATED ALWAYS AS (SUBSTR(b,0,0)) VIRTUAL, ...;
INSERT ...
FLUSH TABLES;
SET sql_mode='';
INSERT ...
But the other way around it crashed:
SET sql_mode='';
CREATE TABLE t1 ... g CHAR(1) GENERATED ALWAYS AS (SUBSTR(b,0,0)) VIRTUAL, ...;
INSERT ...
FLUSH TABLES;
SET sql_mode=ORACLE;
INSERT ...
At CREATE time, SUBSTR was instantiated as Item_func_substr and printed
in the FRM file as substr(). At re-open time with sql_mode=ORACLE, "substr()"
was erroneously instantiated as Item_func_substr_oracle.
Fix:
The fix proposes a symmetric solution. It provides a way to re-parse reliably
all sql_mode dependent functions to their original CREATE TABLE time meaning,
no matter what the open-time sql_mode is.
We take advantage of the same idea we previously used to resolve sql_mode
dependent data types.
Now all sql_mode dependent functions are printed by SHOW using a schema
qualifier when the current sql_mode differs from the function sql_mode:
SET sql_mode='';
CREATE TABLE t1 ... SUBSTR(a,b,c) ..;
SET sql_mode=ORACLE;
SHOW CREATE TABLE t1; -> mariadb_schema.substr(a,b,c)
SET sql_mode=ORACLE;
CREATE TABLE t2 ... SUBSTR(a,b,c) ..;
SET sql_mode='';
SHOW CREATE TABLE t1; -> oracle_schema.substr(a,b,c)
Old replacement names like substr_oracle() are still understood for
backward compatibility and used in FRM files (for downgrade compatibility),
but they are not printed by SHOW any more.
XA support for online alter was totally missing.
Tying on binlog_hton made this hardly visible: simply having binlog_commit
called from xa_commit made an impression that it will automagically work
for online alter, which turns out wrong: all binlog does is writes
"XA END" into trx cache and flushes it to a real binlog.
In comparison, online alter can't do the same, since online replication
happens in a single transaction.
Solution: make a dedicated XA support.
* Extend struct xid_t with a pointer to Online_alter_cache_list
* On prepare: move online alter cache from THD::ha_data to XID passed
* On XA commit/rollback: use the online alter cache stored in this XID.
This makes us pass xid_cache_element->xid to xa_commit/xa_rollback
instead of lex->xid
* Use manual memory management for online alter cache list, instead of
mem_root allocation, since we don't have mem_root connected to the XA
transaction.
There where two errors left from the previous fix.
- subselect.test assumes that mysql.slow_log is empty. This was not
enforced.
- subselect.test dropped a file that does not exists (for safety).
This was fixed by ensuring we don't get a warning if the file does
not exist.
as first character in key
Analysis:
While parsing the path, if '-' is encountered as a part of the key,
the state of the parser changes to error. Hence NULL is returned eventually.
Fix:
If '-' encountered as part of the key, change the state appropriately to
continue scanning the key.
Assertion `!writer.checksum_len || writer.remains == 0' fails upon
concurrent online ALTER and transactions with failing statements and binary
log enabled.
Also another assertion, `pos != (~(my_off_t) 0)', fails in my_seek, upon
reinit_io_cache, on a simplified test. This means that IO_CACHE wasn't
properly initialized, or had an error before.
The overall problem is a deep interference with the effect of an installed
binlog_hton: the assumption about that thd->binlog_get_cache_mngr() is,
sufficiently, NULL, when we shouldn't run the binlog part of
binlog_commit/binlog_rollback, is wrong: as turns out, sometimes the binlog
handlerton can be not installed in current thd, but binlog_commit can be
called on behalf of binlog, as in the bug reported.
One separate condition found is XA recovery of the orphaned transaction,
when binlog_commit is also called, but it has nothing to do with
online alter.
Solution:
Extract online alter operations into a separate handlerton.
1032 (Can't find record) could be emitted when ALTER TABLE is execued vs
concurrent DELETE/UPDATE/other DML that would require search on the online
ALTER's side.
Innodb's INPLACE, in comparison, creates a new trx_t and uses it in scope
of the alter table context.
ALTER TABLE class of statements (i.g. CREATE INDEX, OPTIMIZE, etc.) is
expected to be unaffected by the value of current session's transaction
isolation.
This patch save-and-restores thd->tx_isolation and sets in to
ISO_REPEATABLE_READ for almost a whole mysql_alter_table duration, to avoid
any possible side-effect of it. This should be primarily done before the
lock_tables call, to initialize the storage engine's local value correctly
during the store_lock() call.
sql_table.cc: set thd->tx_isolation to ISO_REPEATABLE_READ in
mysql_alter_table and then restore it to the original value in the end of
the call.
from Item::val_json, UBSAN: member access within null pointer of
type 'struct String' in sql/item_jsonfunc.cc
Analysis:
The first argument of json_schema_valid() needs to be a constant.
Fix:
Parse the schema if the item is constant otherwise set it to return null.
data from a table similar to other JSON functions
Analysis:
Since we are fetching values for every row ( because we are running SELECT
for all rows of a table ), correct value can be only obtained at the time of
calling val_int() because it is called to get value for each row.
Fix:
Set up hash for each row instead of doing it during fixing fields.
Dyncol functions like column_create() encode the
current character set inside the value.
So they cannot be used with --view-protocol.
This patch changes only --disable_view_protocol to
--disable_service_connection.
This bug affected only multi-table update statements and in very rare
cases: one of the tables used at the top level of the statement must be
a derived table containg a row construct with a subquery including hanging
CTE.
Before this patch was applied the function prepare_unreferenced() of the
class With_element when invoked for the the hangin CTE did not properly
restored the value of thd->lex->context_analysis_only. As a result it
became 0 after the call of this function.
For a query affected by the bug this function is called when
JOIN::prepare() is called for the subquery with a hanging CTE. This happens
when Item_row::fix_fields() calls fix_fields() for the subquery. Setting
the value of thd->lex->context_analysis_only forces the caller function
Item_row::fix_fields() to invoke the virtual method is_null() for the
subquery that leads to execution of it. It causes an assertion failure
because the call of Item_row::fix_fields() happens during the invocation
of Multiupdate_prelocking_strategy::handle_end() that calls the function
mysql_derived_prepare() for the derived table used by the UPDATE at the
time when proper locks for the statement tables has not been acquired yet.
With this patch the value of thd->lex->context_analysis_only is restored
to CONTEXT_ANALYSIS_ONLY_DERIVED that is set in the function
mysql_multi_update_prepare().
Approved by Oleksandr Byelkin <sanja@mariadb.com>
Item_char_typecast::print() did not print the "binary" keyword
in such cases:
CAST('a' AS CHAR CHARACTER SET latin1 BINARY)
This caused a difference in "mtr" vs "mtr --view-protocol"
This patch corrects the fix for MDEV-32369. No Item_direct_ref_to_item
objects should be allocated at the optimizer phase after permanent
rewritings have been done.
The patch also adds another test case for MDEV-32369 that uses MyISAM
with more than one row.
Approved by Rex Johnston <rex.johnston@mariadb.com>
- Prevent opening of any user tables in case `upgrade-system-table`
option is used.
- Still there may be uninstalled data types in `mysql` system table so
allow it to perform.
- Closes PR #2790
- Reviewer: <daniel@mariadb.org>, <vicentiu@mariadb.org>
This commit addresses column naming issues with CTEs in the use of prepared
statements and stored procedures. Usage of either prepared statements or
procedures with Common Table Expressions and column renaming may be affected.
There are three related but different issues addressed here.
1) First execution issue. Consider the following
prepare s from "with cte (col1, col2) as (select a as c1, b as c2 from t
order by c1) select col1, col2 from cte";
execute s;
After parsing, items in the select are named (c1,c2), order by (and group by)
resolution is performed, then item names are set to (col1, col2).
When the statement is executed, context analysis is again performed, but
resolution of elements in the order by statement will not be able to find c1,
because it was renamed to col1 and remains this way.
The solution is to save the names of these items during context resolution
before they have been renamed. We can then reset item names back to those after
parsing so first execution can resolve items referred to in order and group by
clauses.
2) Second Execution Issue
When the derived table contains more than one select 'unioned' together we could
reasonably think that dealing with only items in the first select (which
determines names in the resultant table) would be sufficient. This can lead to
a different problem. Consider
prepare st from "with cte (c1,c2) as
(select a as col1, sum(b) as col2 from t1 where a > 0 group by col1
union select a as col3, sum(b) as col4 from t2 where b > 2 group by col3)
select * from cte where c1=1";
When the optimizer (only run during the first execution) pushes the outside
condition "c1=1" into every select in the derived table union, it renames the
items to make the condition valid. In this example, this leaves the first item
in the second select named 'c1'. The second execution will now fail 'group by'
resolution.
Again, the solution is to save the names during context analysis, resetting
before subsequent resolution, but making sure that we save/reset the item
names in all the selects in this union.
3) Memory Leak
During parsing Item::set_name() is used to allocate memory in the statement
arena. We cannot use this call during statement execution as this represents
a memory leak. We directly set the item list names to those in the column list
of this CTE (also allocated during parsing).
Approved by Igor Babaev <igor@mariadb.com>
For some reason, in embedded server, a command
let $a=`$query`
ignores local context. Make a workaround: use SET STATEMENT to set
debug_dbug in the same statement.
ref->null_rejecting is a key_part_map. we need to check
the bit corresponding to the particular store_key.
Note that there are no store_key objects for const ref parts.
This patch fixes a performance regression introduced in the patch for the
bug MDEV-21104. The performance regression could affect queries for which
join buffer was used for an outer join such that its on expression from
which a conjunctive condition depended only on outer tables can be
extracted. If the number of records in the join buffer for which this
condition was false greatly exceeded the number of other records the
slowdown could be significant.
If there is a conjunctive condition extracted from the ON expression
depending only on outer tables this condition is evaluated when interesting
fields of each survived record of outer tables are put into the join buffer.
Each such set of fields for any join operation is supplied with a match
flag field used to generate null complemented rows. If the result of the
evaluation of the condition is false the flag is set to MATCH_IMPOSSIBLE.
When looking in the join buffer for records matching a record of the
right operand of the outer join operation the records with such flags
are not needed to be unpacked into record buffers for evaluation of on
expressions.
The patch for MDEV-21104 fixing some problem of wrong results when
'not exists' optimization by mistake broke the code that allowed to
ignore records with the match flag set to MATCH_IMPOSSIBLE when looking
for matching records. As a result such records were unpacked for each
record of the right operand of the outer join operation. This caused
significant execution penalty in some cases.
One of the test cases added in the patch can be used only for demonstration
of the restored performance for the reported query. The second test case is
needed to demonstrate the validity of the fix.
remove the hack where NO_DEFAULT_VALUE_FLAG was temporarily removed
from a field to initialize DEFAULT() functions in CHECK constraints
while disabling self-reference field checks.
Instead, initialize DEFAULT() functions in CHECK explicitly,
don't call check_field_expression_processor() for CHECK at all.
...errors, then failing ASSERT.
UPDATE queries treat warnings as errors. In this case, an invalid
condition "datetime_key_col >= '2012-01'" caused warning-as-error inside
SQL_SELECT::test_quick_select().
The code that called test_quick_select() ignored this error and continued
join optimization. Then it eventually reached a thd->is_error() check
and failed to setup SJ-Materialization which failed an assert.
Fixed this by making SQL_SELECT::test_quick_select() return error in
its return value, and making any code that calls it to check for error
condition and abort the query if the error is returned.
Places in the code that didn't check for errors from
SQL_SELECT::test_quick_select but now do:
- get_quick_record_count() call in make_join_statistics(),
- test_if_skip_sort_order(),
- "Range checked for each record" code.
Extra error handling fixes and commit text wording by Sergei Petrunia,
Reviewed-by: Sergei Petrunia, Oleg Smirnov
test_if_skip_sort_order() should catch the join types JT_EQ_REF,
JT_CONST and JT_SYSTEM and skip sort order for these.
Such join types imply retrieving of a single row of data, and sorting
of a single row can always be skipped.
Also fixes: MDEV-30050 Inconsistent results of DISTINCT with NOPAD
Problem:
Key segments for CHAR columns where compared using strnncollsp()
for engines MyISAM and Aria.
This did not work correct in case if the engine applyied trailing
space compression.
Fix:
Replacing ha_compare_text() calls to new functions:
- ha_compare_char_varying()
- ha_compare_char_fixed()
- ha_compare_word()
- ha_compare_word_prefix()
- ha_compare_word_or_prefix()
The code branch corresponding to comparison of CHAR column keys
(HA_KEYTYPE_TEXT segment type) now uses ha_compare_char_fixed()
which calls strnncollsp_nchars().
This patch does not change the behavior for the rest of the code:
- comparison of VARCHAR/TEXT column keys
(HA_KEYTYPE_VARTEXT1, HA_KEYTYPE_VARTEXT2 segments types)
- comparison in the fulltext code
Changing the code handling sql_mode-dependent function DECODE():
- removing parser tokens DECODE_MARIADB_SYM and DECODE_ORACLE_SYM
- removing the DECODE() related code from sql_yacc.yy/sql_yacc_ora.yy
- adding handling of DECODE() with help of a new Create_func_func_decode
standard table KEY_COLUMN_USAGE should only show keys where
a user has some privileges on every column of the key
standard table TABLE_CONSTRAINTS should show tables where
a user has any non-SELECT privilege on the table or on any column
of the table
standard table REFERENTIAL_CONSTRAINTS is defined in terms of
TABLE_CONSTRAINTS, so the same rule applies. If the user
has no rights to see the REFERENCED_TABLE_NAME value, it should be NULL
SHOW INDEX (and STATISTICS table) is non-standard, but it seems
reasonable to use the same logic as for KEY_COLUMN_USAGE.
Backporting a part of MDEV-32026 (which also fixed MDEV-32025 in 11.3)
from 11.3 to 10.4.
The reported crash happened with --lower-case-table-names=2
on statements like:
ALTER DATABASE Db1 DEFAULT CHARACTER SET utf8;
ALTER DATABASE `#mysql50#D+b1` UPGRADE DATA DIRECTORY NAME;
lock_schema_name() expects a normalized database name
and assert if a non-normalized name comes.
mysql_alter_db_internal() and mysql_upgrade_db() get
a non-normalized database name in the parameter.
Fixing them to normalize the database name before passing
it to lock_schema_name().
(Variant#3: Allow cross-charset comparisons, use a special
CHARSET_INFO to create lookup keys. Review input addressed.)
Equalities that compare utf8mb{3,4}_general_ci strings, like:
WHERE ... utf8mb3_key_col=utf8mb4_value (MB3-4-CMP)
can now be used to construct ref[const] access and also participate
in multiple-equalities.
This means that utf8mb3_key_col can be used for key-lookups when
compared with an utf8mb4 constant, field or expression using '=' or
'<=>' comparison operators.
This is controlled by optimizer_switch='cset_narrowing=on', which is
OFF by default.
IMPLEMENTATION
Item value comparison in (MB3-4-CMP) is done using utf8mb4_general_ci.
This is valid as any utf8mb3 value is also an utf8mb4 value.
When making index lookup value for utf8mb3_key_col, we do "Charset
Narrowing": characters that are in the Basic Multilingual Plane (=BMP) are
copied as-is, as they can be represented in utf8mb3. Characters that are
outside the BMP cannot be represented in utf8mb3 and are replaced
with U+FFFD, the "Replacement Character".
In utf8mb4_general_ci, the Replacement Character compares as equal to any
character that's not in BMP. Because of this, the constructed lookup value
will find all index records that would be considered equal by the original
condition (MB3-4-CMP).
Approved-by: Monty <monty@mariadb.org>
Problem was that JOIN_TAB::cleanup() was not run because
JOIN::top_join_tab_count was not set in case of early errors.
Fixed by setting JOIN::tab_join_tab_count when JOIN_TAB's are allocated.
Something that should eventually be fixed:
- Cleaning up JOIN_TAB's is now done in 3 different loops.
JOIN_TAB::cleanup() is only doing a partial cleanup. Other cleanups
are done outside of JOIN_TAB::cleanup().
The above should be fixed so that JOIN_TAB::cleanup() is freeing
everything related to it's own memory, including all its sub JOIN_ TAB's.
JOIN::cleanup() should only loop over all it's top JOIN_TAB's and call
JOIN_TAB::cleanup() on these.
This will greatly simplify and speedup the current code (as we now do some
cleanup's twice).
Other usage if persistent statistics is checking 'stats_is_read' in
caller, which is why this was not noticed earlier.
Other things:
- Simplified no_stat_values_provided
