(Variant 2)
Multi-table UPDATE ... ORDER BY ... LIMIT could update the wrong rows when
ORDER BY was resolved by Using temporary + Using filesort.
== Background: ref_pointer_array ==
join->order[->next*]->item point into join->ref_pointer_array, which
has pointers to the used Item objects.
This indirection is employed so that we can switch the ORDER BY expressions
from using the original Items to using the values of their "image" fields
in the temporary table.
The variant of ref_pointer_array that has pointers to temp table fields
is created when JOIN::make_aggr_tables_info() calls
change_refs_to_tmp_fields().
== The problem ==
The created array didn't match element-by-element the original
ref_pointer_array. When arrays were switched, ORDER BY elements started
to point to the wrong temp.table fields, causing the wrong sorting.
== The cause ==
The cause is JOIN::add_fields_for_current_rowid(). This function is
called for UPDATE statements to make the rowids of rows in the original
tables to be saved in the temporary tables.
It adds extra columns to the select list in table_fields argument.
However, select lists are organized in a way that extra elements must
be added *to the front* of the list, and then change_refs_to_tmp_fields()
will add extra fields *to the end* of ref_pointer_array.
So, add_fields_for_current_rowid() adds new fields to the back of
table_fields list. This caused change_refs_to_tmp_fields() to produce
ref_pointer_array slice with extra elements in the front, causing any
references through ref_pointer_array to come to the wrong values.
== The fix ==
Make JOIN::add_fields_for_current_rowid() add fields to the front of
the select list.
We now allow multitable queries with order by and limit, such as:
delete t1.*, t2.* from t1, t2 order by t1.id desc limit 3;
To predict what rows will be deleted, run the equivalent select:
select t1.*, t2.* from t1, t2 order by t1.id desc limit 3;
Additionally, index hints are now supported with single table delete statements:
delete from t2 use index(xid) order by (id) limit 2;
This approach changes the multi_delete SELECT result interceptor to use a temporary
table to collect row ids pertaining to the rows that will be deleted, rather than
directly deleting rows from the target table(s). Row ids are collected during
send_data, then read during send_eof to delete target rows. In the event that the
temporary table created in memory is not big enough for all matching rows, it is
converted to an aria table.
Other changes:
- Deleting from a sequence now affects zero rows instead of emitting an error
Limitations:
- The federated connector does not create implicit row ids, so we to use a key
when conditionally deleting. See the change in federated_maybe_16324629.test
This patch includes a few changes to make the code easier to maintain:
- Renamed SQL_I_List::link_in_list to SQL_I_List::insert. link_in_list was
ambiguous as it could refer to a link or it could refer to a node
- Remove field_name local variable in multi_update::initialize_tables because
it is not used when creating the temporary tables
- multi_update changes:
- Move temp table callocs to init, a more natural location for them, and moved
tables_to_update to const member variable so we don't recompute it.
- Filter out jtbm tables and tables not in the update map, pushing those that
will be updated into an update_targets container. This simplifies checks and
loops in initialize_tables.
After the fix in the previous commit, we should never end up in a
situation where select_lex->first_cond_optimization=false (i.e, "it's done")
but select_lex->leaf_tables_saved==false ("not done").
So, in setup_tables(), revert the the condition added in the fix for
MDEV-25008. Add an assert instead.
Alternative, more general fix, Variant 2.
The problem was as follows: Suppose we are running a PS/SP statement and
we get an error while doing optimization that is done once per statement
life. This may leave the statement data structures in an undefined state,
where it is not safe to execute it again.
The fix: introduce LEX::needs_reprepare and set it in such cases.
Make PS and SP runtime check it and re-prepare the statement before
executing it again.
We do not use Reprepare_observer, because it turns out it is tightly tied
to watching versions of statement's objects. For example, it must not be
used when running the statement for the first time, exactly when the
once-per-statement-lifetime optimizations are done.
InnoDB is limited to row-logging when transaction isolation level is
READ COMMITTED or READ UNCOMMITTED. This limitation is enforced by
a check within ha_innobase::external_lock().
InnoDB also expects number of "successful external locks" to match
number of "external unlocks", which is controlled by the assertion
in subject.
However "unlock" was called even after failing "lock" for high-level
indexes.
Fixed by calling "unlock" only after "successful lock".
Fix a regression that caused assertion thd->is_error() after
sync wait failures. If wsrep_sync_wait() fails make sure a appropriate
error is set. Partially revert 75dd0246f8.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
Problem was incorrect handling of partitioned tables,
because db_type == DB_TYPE_PARTITION_DB
wsrep_should_replicate_ddl incorrectly marked
DDL as not replicatable. However, in partitioned
tables we should check implementing storage engine
from table->file->partition_ht() if available because
if partition handler is InnoDB all DDL should be allowed
even with wsrep_strict_ddl. For other storage engines
DDL should not be allowed and error should be issued.
This is 10.6 version of the fix.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
If wsrep_mode=BINLOG_ROW_FORMAT_ONLY wsrep_forced_binlog_format
can be DEFAULT (UNSPECIFIED) or ROW.
Finally, if wsrep_mode=[REPLICATE_MYISAM|REPLICATE_ARIA] or
wsrep_replicate_myisam=ON we allow wsrep_forced_binlog_format
to be [DEFAULT|ROW].
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
Problem was incorrect handling of partitioned tables,
because db_type == DB_TYPE_PARTITION_DB
wsrep_should_replicate_ddl incorrectly marked
DDL as not replicatable. However, in partitioned
tables we should check implementing storage engine
from table->file->partition_ht() if available because
if partition handler is InnoDB all DDL should be allowed
even with wsrep_strict_ddl. For other storage engines
DDL should not be allowed and error should be issued.
This is 10.5 version of the fix.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
Problem was incorrect condition on wsrep_check_sequence
when ENGINE!=InnoDB.
Fix is not use DB_TYPE_XXX because it is not correct
on dynamic storage engines. Instead used storage engine
name is looked from thd->lex->m_sql_cmd->option_storage_engine_name.
For CREATE TABLE allow anyting except ENGINE=SEQUENCE.
For CREATE SEQUENCE only ENGINE=InnoDB is supported.
For ALTER TABLE if original table contains sequence information
only ENGINE=InnoDB is supported.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
Update all integer columns of SHOW REPLICA STATUS (technically
INFORMATION_SCHEMA.SLAVE_STATUS) to unsigned because, well, they are (:.
Some `uint32` ones were accidentally using the `Field::store(double nr)`
overload because they forgot the `, true` for
`Field::store(longlong nr, bool unsigned_val)`.
The mistake’s harmless, fortunately, as `double` supports over 15
significant decimal digits, well over `uint32`’s 9-and-a-half.
Resize the types and widths of SHOW REPLICA STATUS
(technically `INFORMATION_SCHEMA.SLAVE_STATUS`)
columns to better match their possible values
In case of intentionally but absurdly long lists,
text columns that list an uncapped number of elements
have expanded to accept as many bytes as we could support.
Particularly, the first-gen `Replicate_` filters were
incorrectly typed as singlular `Name()`s during MDEV-33526.
Under `Name`s’ 64-char limit, they could overflow
(read: truncate) even before their lengths got absurd.
In response to `‘MAX_SLAVE_ERRMSG’ was not declared in this scope` in
Embedded builds, a new `#ifdef HAVE_REPLICATION` guard wraps
`slave_status_info` to skip this unused data in Replication-less builds.
For testing, this commit forward-ports a modified cherry-pick of #3795
(the latter targets our oldest maintained LTS as part of MDEV-35948).
> Assert that 1st-gen `replicate_*` filter variables display
> their input – including long but reasonable lists –
> correctly (without truncation) in
> * direct SELECT
> * [semi-new] INFORMATION_SCHEMA.GLOBAL_VARIABLES.VARIABLE_VALUE
> * [new] SHOW REPLICA STATUS
Reviewed-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Implement default values for parameters of stored routines
in both default and oracle mode.
- Default values for cursor parameters are *NOT* supported yet.
- An IN parameter with DEFAULT followed by an OUT param is not supported yet.
This combination will be enabled together with the arrow syntax:
sp1(v=>p2)
The default values can be either literals or expressions.
When it is an expression, it is only evaluated if the parameter
has not been supplied by the caller
(important if the expression has side effects).
(Variant 2)
Multi-table UPDATE ... ORDER BY ... LIMIT could update the wrong rows when
ORDER BY was resolved by Using temporary + Using filesort.
== Background: ref_pointer_array ==
join->order[->next*]->item point into join->ref_pointer_array, which
has pointers to the used Item objects.
This indirection is employed so that we can switch the ORDER BY expressions
from using the original Items to using the values of their "image" fields
in the temporary table.
The variant of ref_pointer_array that has pointers to temp table fields
is created when JOIN::make_aggr_tables_info() calls
change_refs_to_tmp_fields().
== The problem ==
The created array didn't match element-by-element the original
ref_pointer_array. When arrays were switched, ORDER BY elements started
to point to the wrong temp.table fields, causing the wrong sorting.
== The cause ==
The cause is JOIN::add_fields_for_current_rowid(). This function is
called for UPDATE statements to make the rowids of rows in the original
tables to be saved in the temporary tables.
It adds extra columns to the select list in table_fields argument.
However, select lists are organized in a way that extra elements must
be added *to the front* of the list, and then change_refs_to_tmp_fields()
will add extra fields *to the end* of ref_pointer_array.
So, add_fields_for_current_rowid() adds new fields to the back of
table_fields list. This caused change_refs_to_tmp_fields() to produce
ref_pointer_array slice with extra elements in the front, causing any
references through ref_pointer_array to come to the wrong values.
== The fix ==
Make JOIN::add_fields_for_current_rowid() add fields to the front of
the select list.
max_key_length applies only to PRIMARY/UNIQUE/MULTIPLE keys,
but not to FULLTEXT/SPATIAL/VECTOR keys.
this fixes main.partition_geometries test
followup for ecaedbe299
normalize_cond() translated `WHERE col` into `WHERE col<>0`
But the opetator "not equal to 0" does not necessarily exists
for all data types.
For example, the query:
SELECT * FROM t1 WHERE inet6col;
was translated to:
SELECT * FROM t1 WHERE inet6col<>0;
which further failed with this error:
ERROR : Illegal parameter data types inet6 and bigint for operation '<>'
This patch changes the translation from `col<>0` to `col IS TRUE`.
So now
SELECT * FROM t1 WHERE inet6col;
gets translated to:
SELECT * FROM t1 WHERE inet6col IS TRUE;
Details:
1. Implementing methods:
- Field_longstr::val_bool()
- Field_string::val_bool()
- Item::val_int_from_val_str()
If the input contains bad data,
these methods raise a better error message:
Truncated incorrect BOOLEAN value
Before the change, the error was:
Truncated incorrect DOUBLE value
2. Fixing normalize_cond() to generate Item_func_istrue/Item_func_isfalse
instances instead of Item_func_ne/Item_func_eq
3. Making Item_func_truth sargable, so it uses the range optimizer.
Implementing the following methods:
- get_mm_tree(), get_mm_leaf(), add_key_fields() in Item_func_truth.
- get_func_mm_tree(), for all Item_func_truth descendants.
4. Implementing the method negated_item() for all Item_func_truth
descendants, so the negated item has a chance to be sargable:
For example,
WHERE NOT col IS NOT FALSE -- this notation is not sargable
is now translated to:
WHERE col IS FALSE -- this notation is sargable
disable the assert.
also, use the same check for check_that_all_fields_are_given_values()
as it's used in not_null_fields_have_null_values() - to avoid
issuing the same warning twice.
Wsrep_commit_empty happens too early when wsrep is disabled. Let the
cleanup happen at end of statement.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
Fix wrong assertion: function wsrep_restore_kill_after_commit()
asserts `WSREP(thd)`, however for the caller (wsrep_after_statement())
it is enough that the THD has an active transaction. Fixed the
assertion accordingly.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
This commit fixes a bug in MSAN verification in debug builds
related to the member of a lex->definer structure field that may
be uninitialized for some SET statements. The bug is caused by
debug output in a wsrep-specific insert and does not affect
server functionality.
Issue:
Mariadb Galera cluster fails to replicate from
Mysql 5.7 when configured with MASTER_USE_GTID=no
option for CHANGE MASTER.
HOST: mysql, mysql 5.7.44 binlog_format=ROW
HOST: m1, mariadb 10.6 GALERA NODE replicating from HOST mysql, Using_Gtid: No (log file and position)
HOST: m2 mariadb 10.6 GALERA NODE
HOST: m3 mariadb 10.6 GALERA NODE
Error on m1:
2024-05-22 16:11:07 1 [ERROR] WSREP: Vote 0 (success) on 78cebda7-1876-11ef-896b-8a58fca50d36:2565 is inconsistent with group. Leaving cluster.
Error on m2 and m3:
2024-05-22 16:11:06 2 [ERROR] Error in Log_event::read_log_event(): 'Found invalid event in binary log', data_len: 42, event_type: -94
2024-05-22 16:11:06 2 [ERROR] WSREP: applier could not read binlog event, seqno: 2565, len: 482
It fails in Gtid_log_event::is_valid() check on
secondary node when sequence number sent from primary
is 0. On primary for applier or slave thread sequence
number is set to 0, when both thd->variables.gtid_seq_no
and thd->variables.wsrep_gtid_seq_no have value 0.
Solution:
Skip adding Gtid Event on primary for applier or
slave thread when both thd->variables.gtid_seq_no
and thd->variables.wsrep_gtid_seq_no have value 0.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
The Count_handler (derived from Internal_error_handler) used in
Type_handler_datetime_common::convert_item_for_comparison()
suppressed all error codes. This caused a DBUG_ASSERT(thd->is_error())
in Field::sp_prepare_and_store_item() when the error happened
while evaluating a subselect in a scenario like this:
CREATE FUNCTION f() RETURNS INT DETERMINISTIC RETURN (SELECT a FROM t);
CREATE TABLE t (c TIMESTAMP);
SELECT * FROM t WHERE c=f();
Notice there is no such column 'a' in in the table 't'.
Fix:
Handle only DATETIME->TIMESTAMP conversion related errors in Count_handler:
- ER_TRUNCATED_WRONG_VALUE
- ER_DATETIME_FUNCTION_OVERFLOW
thus let other error kinds be processed in their usual way.
The reported scenario now returns this (expected) error:
ERROR 1054 (42S22): Unknown column 'a' in 'SELECT'
Implementation of this task adds ability to raise the signal with
SQLSTATE '02TRG' from a BEFORE INSERT/UPDATE/DELETE trigger and handles
this signal as an indicator meaning 'to throw away the current row'
on processing the INSERT/UPDATE/DELETE statement. The signal with
SQLSTATE '02TRG' has special meaning only in case it is raised inside
BEFORE triggers, for AFTER trigger's this value of SQLSTATE isn't treated
in any special way. In according with SQL standard, the SQLSTATE class '02'
means NO DATA and sql_errno for this class is set to value
ER_SIGNAL_NOT_FOUND by current implementation of MariaDB server.
Implementation of this task assigns the value ER_SIGNAL_SKIP_ROW_FROM_TRIGGER
to sql_errno in Diagnostics_area in case the signal is raised from a trigger
and SQLSTATE has value '02TRG'.
To catch signal with SQLTSATE '02TRG' and handle it in special way, the methods
Table_triggers_list::process_triggers
select_insert::store_values
select_create::store_values
Rows_log_event::process_triggers
and the overloaded function
fill_record_n_invoke_before_triggers
were extended with extra out parameter for returning the flag whether
to skip the current values being processed by INSERT/UPDATE/DELETE
statement. This extra parameter is passed as nullptr in case of AFTER trigger
and BEFORE trigger this parameter points to a variable to store a marker
whether to skip the current record or store it by calling write_record().
This bug prevented building conditions that could be pushed into a derived
table if the derived table was used in a query of a stored procedure and
the conditions contained local variables of the procedure. This could lead
to a slow execution of the procedure.
Also in some cases the bug prevented building conditions that could be
pushed from the HAVING condition into the WHERE condition of a query if
the conditions to be built used local variables of a stored procedure.
To failure to build such pushable conditions was due to lack of a proper
implementation of the virtual method to copy items for the objects of the
class Item_splocal.
Approved by Igor Babaev <igor@mariadb.com>
who had to change the original fix that just added the regular copying of
the nodes of the Item_splocal class to take into account the wrappers
do_get_copy() and do_build_clone() introduced after the fix had been
prepared. He also changed the test case to demonstrate that the fix
was really needed for pushdown from HAVING into WHERE.
Cannot add a foreign key on a table with a long UNIQUE multi-column index, that
contains a foreign key as a prefix.
Check that index algorithms match during the "generated" keys duplicate removal.
mysql_prepare_create_table: Extract a Key initialization part that
relates to length calculation and long unique index designation.
append_system_key_parts call also moves there.
Move this initialization before the duplicate elimination.
Extract WITHOUT OVERPLAPS check into a separate function. It had to be moved
earlier in the code to preserve the order of the error checks, as in the tests.
table->move_fields has some limitations:
1. It cannot be used in cascade
2. It should always have a restoring pair
In this case, an error has occurred before the field ptr was restored, returning
from the function in that state. Even in case of an error, the table can be
reused afterwards and table->field[i]->ptr is not reset in between.
The solution is to restore the field pointers immanently whenever they've been
deviated.
Also add an assertion that ensures that table fields are restored after the use
in close_thread_tables.
(Review input addressed)
After this patch, the optimizer can handle virtual column expressions
in WHERE/ON clauses. If the table has an indexed virtual column:
ALTER TABLE t1
ADD COLUMN vcol INT AS (col1+1),
ADD INDEX idx1(vcol);
and the query uses the exact virtual column expression:
SELECT * FROM t1 WHERE col1+1 <= 100
then the optimizer will be able use index idx1 for it.
This is achieved by walking the WHERE/ON clauses and replacing instances
of virtual column expression (like "col1+1" above) with virtual column's
Item_field (like "vcol"). The latter can be processed by the optimizer.
Replacement is considered (and done) only in items that are potentially
usable to the range optimizer.
When client connections use threadpool, i.e. configuration has:
thread_handling = pool-of-threads it turned out that during wsrep
replication shutdown, not all client connections could be closed.
Reason was that some client threads has stmt_da in state DA_EOF,
and this state was earlier used to detect if client connection
was issuing SHUTDOWN command.
To fix this, the connection executing SHUTDOWN is now detected by
looking at the actual command being executed:
thd->get_command() == COM_SHUTDOWN
During replication shutdown, all other connections but the
SHUTDOWN executor, are terminated.
This commit has new mtr test galera.galera_threadpool, which
opens a number of threadpool client connections, and then
restarts the node to verify that connections in threadpool are
terminated during shutdown.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>
