Commit graph

67717 commits

Author SHA1 Message Date
Seth Shelnutt
23664bc7a5 Fix incorrect DBUG_ENTER message for join_read_last 2019-11-26 12:27:07 +02:00
seppo
38839854b7 MDEV-19572 async slave node fails to apply MyISAM only writes (#1418)
The problem happens when MariaDB master replicates writes for only non InnoDB
tables (e.g. writes to MyISAM table(s)). Async slave node, in Galera cluster,
can apply these writes successfully, but it will, in the end, write gtid position in
mysql.gtid_slave_pos table. mysql.gtid_slave_pos table is InnoDB engine, and
this write makes innodb handlerton part of the replicated "transaction".
Note that wsrep patch identifies that write to gtid_slave_pos should not be replicated
and skips appending wsrep keys for these writes. However, as InnoDB was present
in the transaction, and there are replication events (for MyISAM table) in transaction
cache, but there are no appended keys, wsrep raises an error, and this makes the söave
thread to stop.

The fix is simply to not treat it as an error if async slave tries to replicate a write
set with binlog events, but no keys. We just skip wsrep replication and return successfully.

This commit contains also a mtr test which forces mysql.gtid_slave_pos table isto be
of InnoDB engine, and executes MyISAM only write through asyn replication.

There is additional fix for declaring IO and background slave threads as non wsrep.
These threads should not write anything for wsrep replication, and this is just a safeguard
to make sure nothing leaks into cluster from these slave threads.
2019-11-26 08:49:50 +02:00
willhan
3551cd32a8 MDEV-17508 Fix bug for spider when using "not like"
fix bug for spider where using "not like" (#890)

test case:
t1 is a spider engine table;
  CREATE TABLE `t1` (
  `id` int(11) NOT NULL DEFAULT '0',
  `name` char(64) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=SPIDER

query: "select * from t1 where name  not like 'x%' " would dispatch "select xxx name name like 'x%' " to remote mysqld, is wrong
2019-11-25 15:40:01 +01:00
Vlad Lesin
6718d3bc32 MDEV-21082: isnan/isinf compilation errors, isfinite warnings on MacOS
The fix consists of three commits backported from 10.3:

1) Cleanup isnan() portability checks

(cherry picked from commit 7ffd7fe962)

2) Cleanup isinf() portability checks

Original problem reported by Wlad: re-compilation of 10.3 on top of 10.2
build would cache undefined HAVE_ISINF from 10.2, whereas it is expected
to be 1 in 10.3.

std::isinf() seem to be available on all supported platforms.
(cherry picked from commit bc469a0bdf)

3) Use std::isfinite in C++ code

This is addition to parent revision fixing build failures.

(cherry picked from commit 54999f4e75)
2019-11-19 16:28:15 +03:00
seppo
5c68343db7 MDEV-18497 CTAS async replication from mariadb master crashes galera nodes (#1410)
This PR contains a mtr test for reproducing a failure with replicating create table as select statement (CTAS) through asynchronous mariadb replication to mariadb galera cluster.
The problem happens when CTAS replication contains both create table statement followed by row events for populating the table. In such situation, the galera node operating as mariadb replication slave, will first replicate only the create table part into the cluster, and then perform another replication containing both the create table and row events. This will lead all other nodes to fail for duplicate table create attempt, and crash due to this failure.

PR contains also a fix, which identifies the situation when CTAS has been replicated, and makes further scan in async replication stream to see if there are following row events. The slave node will replicate either single TOI in case the CTAS table is empty, or if CTAS table contains rows, then single bundled write set with create table and row events is replicated to galera cluster.

This fix should keep master server's GTID's for CTAS replication in sync with GTID's in galera cluster.
2019-11-18 15:18:00 +02:00
Varun Gupta
2909725636 MDEV-21044: Wrong result when using a smaller size for sort buffer
Make sure that the sort buffers can store atleast one sort key.
This is needed to make sure that all merge buffers are read else
with no sort keys some merge buffers are skipped because the code
makes a conclusion there is no data to be read.
2019-11-18 11:50:58 +05:30
Andrei Elkin
26fd880d5e manual merge 10.1->10.2 2019-11-11 16:03:43 +02:00
Andrei Elkin
13db50fc03 MDEV-19376 Repl_semi_sync_master::commit_trx assertion failure: ... || !m_active_tranxs->is_tranx_end_pos(trx_wait_binlog_name, trx_wait_binlog_pos)
The assert indicates that the current transaction got caught uncleaned from
the semisync master's cache when it is signaled to proceed upon its
ack receive.

The reason of missed cleanup turns out to be a flaw in the gtid
connect mode.
A submitted by connecting slave value of its last received event's
binlog file *name* was adopted into
{{Repl_semi_sync_master::m_reply_file_name}} as a part of semisync
initialization.

Notice that the initialization still refines the position part of the
submitted last received event's binlog coordinates.
The master side binlog filename:pos refinement is
specific to the gtid connect mode for purpose of computing the latest
binlog file to resume slave feeding from.
Effectively in the gtid connect mode the computed resumption filename:pos
may appear smaller in which case a new post-connect time committing
transaction may be logged with its filename:pos also less than the
submitted coordinates and that triggers the assert.

Fixed with making the semisync initialization to use the refined filename:pos.
It is guaranteed to be less than any new generated transaction's binlog:pos.
2019-11-10 16:16:37 +02:00
Varun Gupta
b1ab2ba583 MDEV-20519: Query plan regression with optimizer_use_condition_selectivity > 1
The issue here is the wrong estimate of the cardinality of a partial join,
the cardinality is too high because the function table_cond_selectivity()
returns an absurd number 100 while selectivity cannot be greater than 1.

When accessing table t by outer reference t1.a via index we do not perform any
range analysis for t. Yet we see TABLE::quick_key_parts[key] and
TABLE->quick_rows[key] contain a non-zero value though these should have been
remained untouched and equal to 0.

Thus real cause of the problem is that TABLE::init does not clean the arrays
TABLE::quick_key_parts[] and TABLE::>quick_rows[].
It should have done it because the TABLE structure created for any
instance of a table can be reused for many queries.
2019-11-07 15:24:21 +05:30
Marko Mäkelä
8688ef22c2 Merge 10.1 to 10.2 2019-11-06 10:18:51 +02:00
Varun Gupta
8afe4bba2e MDEV-20424: New default value for optimizer_use_condition-selectivity leads to bad plan
In the function prev_record_reads where one finds the different row combinations for a
subset of partial join, it did not take into account the selectivity of tables
involved in the subset of partial join.
2019-11-01 17:22:05 +05:30
Robert Bindar
6f86150ab3 MDEV-17896 Assertion `pfs->get_refcount() > 0' failed
Unfortunate DROP TEMPORARY..IF EXISTS on a regular table may allow
subsequent CREATE TABLE statements to steal away the PFS_table_share
instance from the dropped table.
2019-11-01 11:10:04 +02:00
Sergei Golubchik
fd6dfb3b54 Merge branch 'github/10.1' into 10.2 2019-10-30 23:38:05 +01:00
Sergei Golubchik
313855766f cleanup
* use compile_time_assert instead of DBUG_ASSERT

* don't use thd->clear_error(), because
  * the error was already consumed by the error handler, so there is
     nothing to clear
  * it's dangerous to clear errors indiscriminately, if the error came
    from outside of read_statistics_for_tables() it must not be cleared
2019-10-30 23:14:44 +01:00
Sergei Golubchik
5392b4a32c MDEV-20354 All but last insert ignored in InnoDB tables when table locked
mysql_insert() first opens all affected tables (which implicitly
starts a transaction in InnoDB), then stat tables.
A failure to open a stat table caused open_tables() to abort
the current stmt transaction (trans_rollback_stmt()). So, from the
server point of view the following ha_write_row()-s happened outside
of a transactions, and the server didn't bother to commit them.

The server has a mechanism to prevent a transaction being
unexpectedly committed or rolled back in the middle of a statement -
if an operation takes place _in a sub-statement_ it cannot change
the transaction state. Operations on stat tables are exactly that -
they are not allowed to change a transaction state. Put them in
a sub-statement to make sure they don't.
2019-10-30 23:14:44 +01:00
Oleksandr Byelkin
36f67a7dff Merge branch '10.1' into 10.2 2019-10-30 21:33:01 +01:00
Eugene Kosov
b8eff8bce4 avoid accessing TLS 2019-10-30 13:23:15 +03:00
Eugene Kosov
a7e3008ab3 MDEV-20926 UBSAN: load of value 165, which is not a valid value for type bool
Basicaly it's an uninitialized read. 165 is 0xa5 which comes from TRASH_ALLOC()

Fix by calling a class ctor which initializes problematic
TMP_TABLE_PARAM::force_copy_fields field
2019-10-30 12:53:25 +03:00
Jan Lindström
36a9694378 MDEV-18562 [ERROR] InnoDB: WSREP: referenced FK check fail: Lock wait index
Lock wait can happen on secondary index when doing FK checks for wsrep.
We should just return error to upper layer and applier will retry
operation when needed.
2019-10-30 10:14:56 +02:00
Sergei Golubchik
c075c7a861 MDEV-20549 SQL SECURITY DEFINER does not work for INFORMATION_SCHEMA tables
switch to definer privileges when populating I_S tables
2019-10-28 08:17:56 +01:00
Sergei Golubchik
42ada91542 cleanup: RAII helper for swapping of thd->security_ctx 2019-10-28 08:17:56 +01:00
Marko Mäkelä
19ceaf2928 Merge 10.1 into 10.2 2019-10-25 12:57:36 +03:00
Sergei Golubchik
790a74d22b Merge branch 'github/5.5' into 10.1 2019-10-23 15:55:23 +02:00
Monty
7952f7720f MDEV-10748 Server crashes in ha_maria::implicit_commit
Problem was in a combination of LOCK TABLES on several Aria
tables followed by an ALTER TABLE. After the ALTER TABLE there
was a force close + reopen of the alter table. The close failed
because the table was still part of a transaction.

Fixed by calling extra(HA_EXTRA_PREPARE_FOR_FORCED_CLOSE) as
part of closing the table, which ensures that the table is not
anymore part of the current transaction.
2019-10-15 20:26:40 +03:00
Oleksandr Byelkin
4ba763db77 MDEV-13172: Wrong result / SELECT ... WHERE EXISTS ... (with UNIQUE Key)
IS NULL or <=> with unique field does not mean unique row,
because several NULL possible,
so we can not convert to normal join in this case.
2019-10-15 13:20:22 +02:00
Oleksandr Byelkin
c2854c7863 MDEV-20466: SHOW PROCESSLIST truncates query text on \0 bytes
Data should be sent with length.
2019-10-15 09:48:49 +02:00
Eugene Kosov
4d14785546 MDEV-20778 UBSAN: call to function free_rpl_filter() through pointer to incorrect function type
Proper C-style type erasure is done via void*, not via char* or something else.

free_key_cache()
free_rpl_filter(): types were fixed to avoid function pointer type cast which
is still undefined behavior.

Note, that casting from void* to any other pointer type is safe and correct.
2019-10-14 19:16:50 +03:00
Marko Mäkelä
2920377aa0 MDEV-19740: Fix C++11 violations caught by GCC 9.2.1
This is a backport of commit ec28f9532e
to MariaDB Server 10.1.
2019-10-14 16:37:41 +03:00
Marko Mäkelä
24232ec12c Merge 10.1 into 10.2 2019-10-09 08:30:23 +03:00
Sachin Setiya
27664ef29d MDEV-20574 Position of events reported by mysqlbinlog is wrong with encrypted binlogs, SHOW BINLOG EVENTS reports the correct one.
Analysis

Mysqlbinlog output for encrypted binary log
#Q> insert into tab1 values (3,'row 003')
#190912 17:36:35 server id 10221  end_log_pos 980 CRC32 0x53bcb3d3  Table_map: `test`.`tab1` mapped to number 19
# at 940
#190912 17:36:35 server id 10221  end_log_pos 1026 CRC32 0xf2ae5136     Write_rows: table id 19 flags: STMT_END_F

Here we can see Table_map_log_event ends at 980 but Next event starts at 940.
And the reason for that is we do not send START_ENCRYPTION_EVENT to the slave

Solution:-
Send Start_encryption_log_event as Ignorable_log_event to slave(mysqlbinlog),
So that mysqlbinlog can update its log_pos.
Since Slave can request multiple FORMAT_DESCRIPTION_EVENT while master does not
have so We only update slave master pos when master actually have the
FORMAT_DESCRIPTION_EVENT. Similar logic should be applied for START_ENCRYPTION_EVENT.

Also added the test case when new server reads the data from old server which
does not send START_ENCRYPTION_EVENT to slave.

Master Slave Upgrade Scenario.
When Slave is updated first, Slave will have extra logic of handling
START_ENCRYPTION_EVENT But master willnot be sending START_ENCRYPTION_EVENT.
So there will be no issue.
When Master is updated first, It will send  START_ENCRYPTION_EVENT to
slave , But slave will ignore this event in queue_event.
2019-10-08 14:35:34 +05:30
Sergey Vojtovich
adefaeffcc MDEV-19536 - Server crash or ASAN heap-use-after-free in is_temporary_table /
read_statistics_for_tables_if_needed

Regression after 279a907, read_statistics_for_tables_if_needed() was
called after open_normal_and_derived_tables() failure.

Fixed by moving read_statistics_for_tables() call to a branch of
get_schema_stat_record() where result of open_normal_and_derived_tables()
is checked.

Removed THD::force_read_stats, added read_statistics_for_tables() instead.
Simplified away statistics_for_command_is_needed().
2019-10-07 13:30:22 +04:00
Sergey Vojtovich
e43791d4dc Cleanup EITS
Moved EITS allocation inside read_statistics_for_tables_if_needed().
Removed redundant is_safe argument.
2019-10-02 15:23:59 +04:00
Alexander Barkov
f203245e9e Merge remote-tracking branch 'origin/10.1' into 10.2 2019-10-01 07:11:54 +04:00
Alexander Barkov
4bcf524482 Merge remote-tracking branch 'origin/5.5' into 10.1 2019-10-01 06:07:56 +04:00
Alexander Barkov
d481f69a7d MDEV-20704 An index on a double column erroneously uses prefix compression 2019-10-01 05:55:14 +04:00
Marko Mäkelä
46b785262b Fix -Wunused for CMAKE_BUILD_TYPE=RelWithDebInfo
For release builds, do not declare unused variables.

unpack_row(): Omit a debug-only variable from WSREP diagnostic message.

create_wsrep_THD(): Fix -Wmaybe-uninitialized for the PSI_thread_key.
2019-09-30 12:49:53 +03:00
Sujatha
9b80f9300d MDEV-20645: Replication consistency is broken as workers miss the error notification from an earlier failed group.
Analysis:
========
In general if there are three groups.
1 - Inserts 32 which fails due to local entry '32' on slave.
2 - Inserts 33
3 - Inserts 34

Each group considers itself as a waiter and it waits for prior group 'waitee'.
This is done in 'register_wait_for_prior_event_group_commit'. If there is no
other parallel group being scheduled then no waitee will be there.

Let us assume 3 groups are being scheduled in parallel.

3-> waits for 2-> waits for->1

'1' upon completion it checks is there any registered subsequent waiter. If
so it wakes up the subsequent waiter with its execution status. This execution
status is stored in wakeup_error.

If '1' failed then it sends corresponding wakeup_error to 2. Then '2' aborts
and it propagates error to '3'.  So all further commits are aborted.  This
mechanism works only when all transactions reach a stage where they are
waiting for their prior commit to complete.

In case of optimistic following scenario occurs.

1,2,3 are scheduled in parallel.

3 - Reaches group_commit_code waits for 2 to complete.
1 - errors out sets stop_on_error_sub_id=1.

When a group execution results in error its corresponding sub_id is set to
'stop_on_error_sub_id'. Any new groups queued for execution will check if
their sub_id is > stop_on_error_sub_id.  If it is true their execution will be
skipped as prior group execution failed.  'skip_event_group=1' will be set.
Since the execution of SQL thread is about to stop we just skip execution of
all the following event groups.  We still do all the normal waiting and wakeup
processing between the event groups as a simple way to ensure that everything
is stopped and cleaned up correctly.

Upon error '1' transaction checks for registered waiters. Since no one is
there it simply goes away.

2 - Starts the execution. It checks do I have a waitee.

Since wait_commit_sub_id == entry->last_committed_sub_id no waitee is set.

Secondly: 'entry->stop_on_error_sub_id' is set by '1'st execution.  Now
'handle_parallel_thread' code checks if the current group 'sub_id' is greater
than the 'sub_id' set within 'stop_on_error_sub_id'.

Since the above is true 'skip_event_group=true' is set.  Simply call
'wait_for_prior_commit' to wakeup all waiters.  Group '2' didn't had any
waitee and its execution is skipped.  Hence its wakeup_error=0.It sends a
positive wakeup signal to '3'. Which commits. This results in a missed
transaction. i.e 33 is missed and 34 is committed.

Fix:
===
When a worker learns that an earlier transaction execution has failed, and it
should not proceed for further execution, it should mark its own execution
status as failed so that it alerts its followers to abort as well.
2019-09-30 13:22:37 +05:30
Marko Mäkelä
ca9e0089d5 MDEV-19740: Fix GCC 9.2.1 -Wmaybe-uninitialized on AMD64
For CMAKE_BUILD_TYPE=Debug, the default MYSQL_MAINTAINER_MODE=AUTO
implies -Werror along with other flags in cmake/maintainer.cmake,
which would break the debug builds when CMAKE_CXX_FLAGS include -O2.

This fix includes a backport of 6dd3f24090
from MariaDB 10.3.
2019-09-27 10:43:23 +03:00
Alexander Barkov
0a359d7627 Merge remote-tracking branch 'origin/10.1' into 10.2 2019-09-24 19:19:25 +04:00
Alexander Barkov
5f118b26c8 Merge remote-tracking branch 'origin/5.5' into 10.1 2019-09-24 11:18:39 +04:00
Alexander Barkov
67b0faa29e MDEV-20495 Assertion `precision > 0' failed in decimal_bin_size upon CREATE .. SELECT with zerofilled decimal
Also fixes:

MDEV-20560 Assertion `precision > 0' failed in decimal_bin_size upon SELECT with MOD short unsigned decimal

Changing the way how Item_func_mod calculates its max_length.
It now uses decimal_precision(), decimal_scale() and unsigned_flag
of its arguments, like all other Item_num_op descendants do.
2019-09-24 10:46:18 +04:00
Varun Gupta
ab9f378b0b Backporting 273d8eb12c Proper fix for disabling warnings in read_statistics_for_table() 2019-09-23 14:11:48 +05:30
Marko Mäkelä
44c5144943 Merge 10.1 into 10.2 2019-09-23 08:26:08 +03:00
Varun Gupta
896974fc3d MDEV-18094: Query with order by limit picking index scan over filesort
In the function test_if_cheaper_ordering we make a decision if using an index is better than
using filesort for ordering. If we chose to do range access then in test_quick_select we
should make sure that cost for table scan is set to DBL_MAX so that it is not picked.
2019-09-21 12:14:05 +05:30
Anel Husakovic
1ad79c8187 MDEV-19679 - CREATE SERVER needs tweaks for compatibility with CONNECT engine 2019-09-20 01:54:38 -07:00
Marko Mäkelä
bb4214272a Merge 10.1 into 10.2 2019-09-18 16:24:48 +03:00
Varun Gupta
273d8eb12c MDEV-20589: Server still crashes in Field::set_warning_truncated_wrong_value
The flag is_stat_field is not set for the min_value and max_value of field items
inside table share. This is a must requirement as we don't want to throw
warnings of truncation when we read values from the statistics table to the column
statistics of table share fields.
2019-09-18 15:06:02 +05:30
Teemu Ollakka
40beeb1402 MDEV-20561 Galera node shutdown fails in non-Primary (#1386)
Command COM_SHUTDOWN was rejected in non-Primary because
server_command_flags[COM_SHUTDOWN] had value CF_NO_COM_MULTI
instead of CF_SKIP_WSREP_CHECK.

As a fix removed assignment
server_command_flags[CF_NO_COM_MULTI]= CF_NO_COM_MULTI
which overwrote server_command_flags[COM_SHUTDOWN].
2019-09-13 09:18:11 +03:00
Igor Babaev
deb9121fdf MDEV-20576 A new assertion added to check validity of calculated
selectivity values fails

After having set the assertion that checks validity of selectivity values
returned by the function table_cond_selectivity() a test case from
order_by.tesst failed. The failure occurred because range optimizer could
return as an estimate of the cardinality of the ranges built for an index
a number exceeding the total number of records in the table.

The second bug is more subtle. It may happen when there are several
indexes with same prefix defined on the first joined table t accessed by
a constant ref access. In this case the range optimizer estimates the
number of accessed records of t for each usable index and these
estimates can be different. Only the first of these estimates is taken
into account when the selectivity of the ref access is calculated.
However the optimizer later can choose a different index that provides
a different estimate. The function table_condition_selectivity() could use
this estimate to discount the selectivity of the ref access. This could
lead to an selectivity value returned by this function that was greater
that 1.
2019-09-12 23:01:11 -07:00
Sergei Petrunia
c8dc866fde MDEV-20371: Invalid reads at plan refinement stage: join->positions...
best_access_path() is called from two optimization phases:

1. Plan choice phase, in choose_plan(). Here, the join prefix being
   considered is in join->positions[]

2. Plan refinement stage, in fix_semijoin_strategies_for_picked_join_order
   Here, the join prefix is in join->best_positions[]

It used to access join->positions[] from stage #2. This didnt cause any
valgrind or asan failures (as join->positions[] has been written-to before)
but the effect was similar to that of reading the random data:
The join prefix we've picked (in join->best_positions) could have
nothing in common with the join prefix that was last to be considered
(in join->positions).
2019-09-11 17:06:50 +03:00
Marko Mäkelä
f7fe51f126 Merge 10.1 into 10.2 2019-09-11 08:48:00 +03:00
Marko Mäkelä
d6f0e60a67 Merge 5.5 into 10.1
Null merge the MDEV-14383 fix; it affects 5.5 only.
2019-09-11 08:11:46 +03:00
Alexander Barkov
5ec4efb7b1 Adding missing semicolons to sql_yacc.yy (10.2) 2019-09-11 04:44:24 +04:00
Vladislav Vaintroub
2336e0b394 MDEV-20206 : Crash inside timer_callback()[threadpool_win.cc:419]
The most likely cause of the crash is that a timer fired, after it was closed.

MSDN documents such a possibility, in the documentation for
CloseThreadpoolTimer() function, and recommends disabling the timer before
calling WaitForThreadpoolTimerCallbacks()/CloseThreadpoolTimer().

The fix follows this recommendation.

Note, that 5.5-10.1 disabled the timer before close, but this code
was lost in threadpool refactoring in 10.2
2019-09-09 13:52:30 +02:00
Vladislav Vaintroub
803bb5cdf8 Windows, cmake : Fix occasional link error when switching between debug to optimized compilation
For Visual Studio generator, use a per-config .def/.lib files with symbols
exported from mysqld.exe

Functions exported from mysqld.exe may differ between debug/optimized
compilation, e.g dbug functions are missing in release config.
2019-09-09 13:52:30 +02:00
Alexey Botchkov
031c695b8c MDEV-16594 ALTER DATA DIRECTORY in PARTITIONS of InnoDB storage does nothing silently
InnoDB intentionally (it's a documented behavior) ignores changing of
DATA DIRECTORY and INDEX DIRECTORY for partitions. Though we should
issue warning when this happens.
2019-09-09 15:39:12 +04:00
Jan Lindström
0fd5b11eb0 MDEV-20511: Galera replication of events is not consistent
After SST from master node (the one where event is ENABLED) - you will end up with the event enabled on two nodes, hence it's now being executed twice. It can be solved by comparing event's originator with server_id. if not equal, then change its status to 'SLAVESIDE_DISABLED'

Changes to be committed:
	new file:   mysql-test/suite/galera/r/galera_events2.result
	new file:   mysql-test/suite/galera/t/galera_events2.test
	modified:   sql/events.cc
2019-09-09 14:23:59 +03:00
Alexander Barkov
8750df43ab MDEV-20517 Assertion `!is_expensive()' failed in Item::value_depends_on_sql_mode_const_item 2019-09-07 07:44:54 +04:00
Sergei Golubchik
5a9e2b77d4 Part3: MDEV-18156 Assertion 0' failed or btr_validate_index(index, 0, false)' in row_upd_sec_index_entry or error code 126: Index is corrupted upon DELETE with PAD_CHAR_TO_FULL_LENGTH
Don't break compatibility in GA releases.
Warn the user, but don't refuse to create a table until 10.5
2019-09-06 16:35:56 +02:00
Sergei Golubchik
8dca4cf53f MDEV-20403 Assertion 0' or Assertion btr_validate_index(index, 0)' failed in row_upd_sec_index_entry or error code 126: Index is corrupted upon UPDATE with TIMESTAMP..ON UPDATE
remove a special treatment of a bare DEFAULT keyword that made it
behave inconsistently and differently from DEFAULT(column).
Now all forms of the explicit assignment of a default column value
behave identically, and all count as an explicitly assigned value
(for the purpose of ON UPDATE NOW).

followup for c7c481f4d9
2019-09-04 15:37:23 +02:00
Marko Mäkelä
b07beff894 MDEV-15326: InnoDB: Failing assertion: !other_lock
MySQL 5.7.9 (and MariaDB 10.2.2) introduced a race condition
between InnoDB transaction commit and the conversion of implicit
locks into explicit ones.

The assertion failure can be triggered with a test that runs
3 concurrent single-statement transactions in a loop on a simple
table:

CREATE TABLE t (a INT PRIMARY KEY) ENGINE=InnoDB;
thread1: INSERT INTO t SET a=1;
thread2: DELETE FROM t;
thread3: SELECT * FROM t FOR UPDATE; -- or DELETE FROM t;

The failure scenarios are like the following:
(1) The INSERT statement is being committed, waiting for lock_sys->mutex.
(2) At the time of the failure, both the DELETE and SELECT transactions
are active but have not logged any changes yet.
(3) The transaction where the !other_lock assertion fails started
lock_rec_convert_impl_to_expl().
(4) After this point, the commit of the INSERT removed the transaction from
trx_sys->rw_trx_set, in trx_erase_lists().
(5) The other transaction consulted trx_sys->rw_trx_set and determined
that there is no implicit lock. Hence, it grabbed the lock.
(6) The !other_lock assertion fails in lock_rec_add_to_queue()
for the lock_rec_convert_impl_to_expl(), because the lock was 'stolen'.
This assertion failure looks genuine, because the INSERT transaction
is still active (trx->state=TRX_STATE_ACTIVE).

The problematic step (4) was introduced in
mysql/mysql-server@e27e0e0bb7
which fixed something related to MVCC (covered by the test
innodb.innodb-read-view). Basically, it reintroduced an error
that had been mentioned in an earlier commit
mysql/mysql-server@a17be6963f:
"The active transaction was removed from trx_sys->rw_trx_set prematurely."

Our fix goes along the following lines:

(a) Implicit locks will released by assigning
trx->state=TRX_STATE_COMMITTED_IN_MEMORY as the first step.
This transition will no longer be protected by lock_sys_t::mutex,
only by trx->mutex. This idea is by Sergey Vojtovich.
(b) We detach the transaction from trx_sys before starting to release
explicit locks.
(c) All callers of trx_rw_is_active() and trx_rw_is_active_low() must
recheck trx->state after acquiring trx->mutex.
(d) Before releasing any explicit locks, we will ensure that any activity
by other threads to convert implicit locks into explicit will have ceased,
by checking !trx_is_referenced(trx). There was a glitch
in this check when it was part of lock_trx_release_locks(); at the end
we would release trx->mutex and acquire lock_sys->mutex and trx->mutex,
and fail to recheck (trx_is_referenced() is protected by trx_t::mutex).
(e) Explicit locks can be released in batches (LOCK_RELEASE_INTERVAL=1000)
just like we did before.

trx_t::state: Document that the transition to COMMITTED is only
protected by trx_t::mutex, no longer by lock_sys_t::mutex.

trx_rw_is_active_low(), trx_rw_is_active(): Document that the transaction
state should be rechecked after acquiring trx_t::mutex.

trx_t::commit_state(): New function to change a transaction to committed
state, to release implicit locks.

trx_t::release_locks(): New function to release the explicit locks
after commit_state().

lock_trx_release_locks(): Move much of the logic to the caller
(which must invoke trx_t::commit_state() and trx_t::release_locks()
as needed), and assert that the transaction will have locks.

trx_get_trx_by_xid(): Make the parameter a pointer to const.

lock_rec_other_trx_holds_expl(): Recheck trx->state after acquiring
trx->mutex, and avoid a redundant lookup of the transaction.

lock_rec_queue_validate(): Recheck impl_trx->state while holding
impl_trx->mutex.

row_vers_impl_x_locked(), row_vers_impl_x_locked_low():
Document that the transaction state must be rechecked after
trx_mutex_enter().

trx_free_prepared(): Adjust for the changes to lock_trx_release_locks().
2019-09-04 09:42:38 +03:00
Sergei Golubchik
c7c481f4d9 MDEV-20403 Assertion 0' or Assertion btr_validate_index(index, 0)' failed in row_upd_sec_index_entry or error code 126: Index is corrupted upon UPDATE with TIMESTAMP..ON UPDATE
Three issues here:
* ON UPDATE DEFAULT NOW columns were updated after generated columns
  were computed - this broke indexed virtual columns
* ON UPDATE DEFAULT NOW columns were updated after BEFORE triggers,
  so triggers didn't see the correct NEW value
* in case of a multi-update generated columns were also updated
  after BEFORE triggers
2019-09-03 20:34:30 +02:00
Sergei Golubchik
3789692d17 don't compare unassigned columns
on UPDATE, compare_record() was comparing all columns that are marked
for writing. But generated columns that are written to the table are
always deterministic and cannot change unless normal non-generated
columns were changed. So it's enough to compare only non-generated
columns that were explicitly assigned values in the SET clause.
2019-09-03 20:34:30 +02:00
Sergei Golubchik
17ab02f4b0 cleanup: on update default now
* remove one level of virtual functions
* remove redundant checks
* remove an if() as the value is always known at compilation time

don't pretend that "DEFAULT expr" and "ON UPDATE DEFAULT NOW"
are "basically the same thing"
2019-09-03 20:34:30 +02:00
Alexander Barkov
ef00ac4c86 Part2: MDEV-18156 Assertion 0' failed or btr_validate_index(index, 0, false)' in row_upd_sec_index_entry or error code 126: Index is corrupted upon DELETE with PAD_CHAR_TO_FULL_LENGTH
This patch allows the server to open old tables that have
"bad" generated columns (i.e. indexed virtual generated columns,
persistent generated columns) that depend on sql_mode,
for general things like SELECT, INSERT, DROP, etc.
Warning are issued in such cases.

Only these commands are now disallowed and return an error:
- CREATE TABLE introducing a "bad" generated column
- ALTER TABLE introducing a "bad" generated column
- CREATE INDEX introdicing a "bad" generated column
  (i.e. adding an index on a virtual generated column
   that depends on sql_mode).

Note, these commands are allowed:
- ALTER TABLE removing a "bad" generate column
- ALTER TABLE removing an index from a "bad" virtual generated column
- DROP INDEX removing an index from a "bad" virtual generated column
but only if the table does not have any "bad" columns as a result.
2019-09-03 09:51:35 +04:00
Alexander Barkov
dc719597ee MDEV-18156 Assertion 0' failed or btr_validate_index(index, 0, false)' in row_upd_sec_index_entry or error code 126: Index is corrupted upon DELETE with PAD_CHAR_TO_FULL_LENGTH
This change takes into account a column's GENERATED ALWAYS AS
expression dependcy on sql_mode's PAD_CHAR_TO_FULL_LENGTH and
NO_UNSIGNED_SUBTRACTION flags.

Indexed virtual columns as well as persistent generated columns are
now not allowed to have such dependencies to avoid inconsistent data
or index files on sql_mode changes.
So an error is now returned in cases like this:

  CREATE OR REPLACE TABLE t1
  (
    a CHAR(5),
    v VARCHAR(5) AS (a) PERSISTENT -- CHAR->VARCHAR or CHAR->TEXT = ERROR
  );

Functions RPAD() and RTRIM() can now remove dependency on
PAD_CHAR_TO_FULL_LENGTH. So this can be used instead:

  CREATE OR REPLACE TABLE t1
  (
    a CHAR(5),
    v VARCHAR(5) AS (RTRIM(a)) PERSISTENT
  );

Note, unlike CHAR->VARCHAR and CHAR->TEXT this still works,
not RPAD(a) is needed:

  CREATE OR REPLACE TABLE t1
  (
    a CHAR(5),
    v CHAR(5) AS (a) PERSISTENT -- CHAR->CHAR is OK
  );

More sql_mode flags may affect values of generated columns.
They will be addressed separately.

See comments in sql_mode.h for implementation details.
2019-09-03 05:34:53 +04:00
Monty
9cba6c5aa3 Updated mtr files to support different compiled in options
This allows one to run the test suite even if any of the following
options are changed:
- character-set-server
- collation-server
- join-cache-level
- log-basename
- max-allowed-packet
- optimizer-switch
- query-cache-size and query-cache-type
- skip-name-resolve
- table-definition-cache
- table-open-cache
- Some innodb options
etc

Changes:
- Don't print out the value of system variables as one can't depend on
  them to being constants.
- Don't set global variables to 'default' as the default may not
  be the same as the test was started with if there was an additional
  option file. Instead save original value and reset it at end of test.
- Test that depends on the latin1 character set should include
  default_charset.inc or set the character set to latin1
- Test that depends on the original optimizer switch, should include
  default_optimizer_switch.inc
- Test that depends on the value of a specific system variable should
  set it in the test (like optimizer_use_condition_selectivity)
- Split subselect3.test into subselect3.test and subselect3.inc to
  make it easier to set and reset system variables.
- Added .opt files for test that required specfic options that could
  be changed by external configuration files.
- Fixed result files in rockdsb & tokudb that had not been updated for
  a while.
2019-09-01 19:17:35 +03:00
Oleksandr Byelkin
14149d6c33 Merge remote-tracking branch 'connect/10.2' into 10.2 2019-08-30 16:52:43 +02:00
Marko Mäkelä
5f35e103ee Merge 10.1 into 10.2 2019-08-28 15:23:21 +03:00
Oleksandr Byelkin
9cd6e7ad73 MDEV-16932: ASAN heap-use-after-free in my_charlen_utf8 / my_well_formed_char_length_utf8 on 2nd execution of SP with ALTER trying to add bad CHECK
Make automatic name generation during execution (not prepare).

Check result of memory allocation operation.
2019-08-28 12:06:52 +02:00
Sujatha
e7b71e0daa MDEV-19925: Column ... cannot be converted from type 'varchar(20)' to type 'varchar(20)'
Cherry picking:
Bug#25135304: RBR: WRONG FIELD LENGTH IN ERROR MESSAGE
commit 47bd3f7cf3c8518f62b1580ec65af2ba7ac13b95

Description:
============
In row based replication, when replicating from a table with a field with
character set set to UTF8mb3 to the same table with the same field set to
character set UTF8mb4 I get a confusing error message:

For VARCHAR: VARCHAR(1) 'utf8mb3' to VARCHAR(1) 'utf8mb4'
"Column 0 of table 'test.t1' cannot be converted from type 'varchar(3)' to
type 'varchar(1)'"

Similar issue with CHAR type as well.

Issue with respect to BLOB types:

For BLOB: LONGBLOB to TINYBLOB - Error message displays incorrect blob type.
"Column 0 of table 'test.t1' cannot be converted from type 'tinyblob' to type
'tinyblob'"

For BINARY to BINARY - Error message displays incorrect type for master side
field.
"Column 0 of table 'test.t' cannot be converted from type 'char(1)' to type
'binary(10)'"
Similar issue exists for VARBINARY type. It is displayed as 'VARCHAR'.

Analysis:
=========
In Row based replication charset information is not sent as part of metadata
from master to slave.

For VARCHAR field its character length is converted into equivalent
octets/bytes and stored internally. At the time of displaying the data to user
it is converted back to original character length.

For example:
VARCHAR(2)- utf8mb3 is stored as:2*3 = VARCHAR(6)
At the time of displaying it to user
VARCHAR(6)- charset utf8mb3:6/3= VARCHAR(2).

At present the internally converted octect length is sent from master to slave
with out providing the charset information. On slave side if the type
conversion fails 'show_sql_type' function is used to get the type specific
information from metadata. Since there is no charset information is available
the filed type is displayed as VARCHAR(6).

This results in confused error message.

For CHAR fields
CHAR(1)- utf8mb3 - CHAR(3)
CHAR(1)- utf8mb4 - CHAR(4)

'show_sql_type' function which retrieves type information from metadata uses
(bytes/local charset length) to get actual character length. If slave's chaset
is 'utf8mb4' then

CHAR(3/4)-->CHAR(0)
CHAR(4/4)-->CHAR(1).

This results in confused error message.

Analysis for BLOB type issue:

BLOB's length is represented in two forms.
1. Actual length
i.e
  (length < 256) type= MYSQL_TYPE_TINY_BLOB;
  (length < 65536) type= MYSQL_TYPE_BLOB; ...

2. packlength - The number of bytes used to represent the length of the blob
  1- tinyblob
  2- blob ...

In row based replication only the packlength is written in the binary log. On
the slave side this packlength is interpreted as actual length of the blob.
Hence the length is always < 256 and the type is displayed as tiny blob.

Analysis for BINARY to BINARY type issue:
The character set information is needed to identify a filed's type as char or
binary. Since master side character set information is not available on the
slave side both binary and char fields are displayed as char.

Fix:
===
For CHAR and VARCHAR fields display their length in octets for both source and
target fields. For target field display the charset information if it is
relevant.

For blob type changed the code to use the packlength and display appropriate
blob type in error message.

For binary and varbinary fields use the slave side character set as reference
to map them to binary or varbinary fields.
2019-08-27 13:05:04 +05:30
Alexander Barkov
29bbf4749e MDEV-19699 Server crashes in Item_null_result::field_type upon SELECT with ROLLUP on constant table
Also fixes:

MDEV-20431 GREATEST(int_col,date_col) returns wrong results in a view
2019-08-27 09:13:20 +04:00
Julius Goryavsky
de0f93fb0d MDEV-20420: SST failed after MDEV-18863 in some test configurations
After applying MDEV-18863, in some test configurations, SST
may fails due to duplication of some parameters (in particular
"--port") in the main part of the command line and after
"--mysqld-args", as well as due to incorrect interpretation
of the parameter "--port" passed after "--mysqld-args" when
the SST script is invoked without explicitly specifying a port
for SST. In addition, it is necessary to correctly handle spaces,
quotation marks and special characters when copying original
arguments from the argv[] array to a new command line (after
"--mysqld-args"). This patch resolves these shortcomings.
2019-08-26 13:41:06 +02:00
Stephen Long
4e89fdb9d8 MDEV-19837 and MDEV-19816: Change some comments 2019-08-21 02:52:41 -07:00
Marko Mäkelä
48c67038b9 Merge 10.1 into 10.2
For MDEV-15955, the fix in create_tmp_field_from_item() would cause a
compilation error. After a discussion with Alexander Barkov, the fix
was omitted and only the test case was kept.

In 10.3 and later, MDEV-15955 is fixed properly by overriding
create_tmp_field() in Item_func_user_var.
2019-08-20 09:15:28 +03:00
Marko Mäkelä
bc89b1c558 MDEV-18863: Fix -Wsign-compare 2019-08-20 07:47:11 +03:00
Marko Mäkelä
a02dd7e614 Merge 5.5 into 10.1 2019-08-20 07:31:44 +03:00
Julius Goryavsky
ff6d3075d5 MDEV-18863: Galera SST scripts can't read [mysqldN] option groups
Some users and some scripts (for example, mysqld_multi.sh) use special
option groups with names like [mysqld1], [mysqld2], ..., [mysqldN].

But SST scripts can't currently fully support these option groups.
The only option group-related value it gets from the server is
--defaults-group-suffix, if that option was set for mysqld when
the server was started.

However, the SST scripts does not get told by the server to read
these option groups, so this means that the SST script will fail
to read options like innodb-data-home-dir when it is in a option
group like [mysqld1]...[mysqldN].

Moreover, SST scripts ignore many parameters that can be passed
to them explicitly and cannot transfer them further, for example,
to the input of mariabackup utility. Ideally, we want to transfer
all the parameters of the original mysqld call to utilities such
as mariabackup, however the SST script does not receive these
parameters from the server and therefore cannot transfer them to
mariabackup.

To correct these shortcomings, we need to transfer to the scripts
all of the parameters of the original mysqld call, and in the SST
scripts themselves provide for the transfer all of these parameters
to utilities such as mariabackup. To prevent these parameters from
mixing with the script's own parameters, they should be transferred
to SST script after the special option "--mysqld-args", followed by
the string argument with the original parameters, as it received by
the mysqld call at the time of launch (further all these parameters
will be passed to mariabackup, for example).

In addition, the SST scripts themselves must be refined so that
they can read the parameters from the user-selected group, not just
from the global mysqld configuration group. And also so that they
can receive the parameters (which important for their work) as
command-line arguments.
2019-08-19 23:45:35 +02:00
Julius Goryavsky
457dc9d64d MDEV-18863: Galera SST scripts can't read [mysqldN] option groups
Some users and some scripts (for example, mysqld_multi.sh) use special
option groups with names like [mysqld1], [mysqld2], ..., [mysqldN].

But SST scripts can't currently fully support these option groups.
The only option group-related value it gets from the server is
--defaults-group-suffix, if that option was set for mysqld when
the server was started.

However, the SST scripts does not get told by the server to read
these option groups, so this means that the SST script will fail
to read options like innodb-data-home-dir when it is in a option
group like [mysqld1]...[mysqldN].

Moreover, SST scripts ignore many parameters that can be passed
to them explicitly and cannot transfer them further, for example,
to the input of mariabackup utility. Ideally, we want to transfer
all the parameters of the original mysqld call to utilities such
as mariabackup, however the SST script does not receive these
parameters from the server and therefore cannot transfer them to
mariabackup.

To correct these shortcomings, we need to transfer to the scripts
all of the parameters of the original mysqld call, and in the SST
scripts themselves provide for the transfer all of these parameters
to utilities such as mariabackup. To prevent these parameters from
mixing with the script's own parameters, they should be transferred
to SST script after the special option "--mysqld-args", followed by
the string argument with the original parameters, as it received by
the mysqld call at the time of launch (further all these parameters
will be passed to mariabackup, for example).

In addition, the SST scripts themselves must be refined so that
they can read the parameters from the user-selected group, not just
from the global mysqld configuration group. And also so that they
can receive the parameters (which important for their work) as
command-line arguments.
2019-08-19 23:43:16 +02:00
Igor Babaev
e746f451d5 MDEV-20265 Unknown column in field list
This patch corrects the fix of the patch for mdev-19421 that resolved
the problem of parsing some embedded join expressions such as
  t1 join t2 left join t3 on t2.a=t3.a on t1.a=t2.a.
Yet the patch contained a bug that prevented proper context analysis
of the queries where such expressions were used together with comma
separated table references in from clauses.
2019-08-19 14:20:41 -07:00
Alexander Barkov
f987de7122 Merge remote-tracking branch 'origin/5.5' into 10.1 2019-08-16 20:58:14 +04:00
Alexander Barkov
ec1f195ecf MDEV-15955 Assertion `field_types == 0 || field_types[field_pos] == MYSQL_TYPE_LONGLONG' failed in Protocol_text::store_longlong 2019-08-16 14:32:44 +04:00
Jan Lindström
fa74088838 MDEV-18778: mysql_tzinfo_to_sql does not work correctly in MariaDB Galera
There were two problems:

(1) If user wanted same time zone information on all nodes in the Galera
cluster all updates were not replicated as time zone information was
stored on MyISAM tables. This is fixed on Galera by altering time zone
tables to InnoDB while they are modified.

(2) If user wanted different time zone information to nodes in the Galera
cluster TRUNCATE TABLE for time zone tables was replicated by Galera
destroying time zone information from other nodes. This is fixed
on Galera by introducing new option for mysql_tzinfo_to_sql_symlink
tool --skip-write-binlog to disable Galera replication while
time zone tables are modified.

Changes to be committed:
	modified:   mysql-test/r/mysql_tzinfo_to_sql_symlink.result
	modified:   mysql-test/suite/wsrep/r/mysql_tzinfo_to_sql_symlink.result
	new file:   mysql-test/suite/wsrep/r/mysql_tzinfo_to_sql_symlink_skip.result
	new file:   mysql-test/suite/wsrep/t/mysql_tzinfo_to_sql_symlink_skip.test
	modified:   sql/tztime.cc
2019-08-16 07:01:30 +03:00
Marko Mäkelä
ec28f9532e MDEV-19740: Fix C++11 violations caught by GCC 9.2.1 2019-08-15 15:58:37 +03:00
Sergei Petrunia
1c75ad6eed MDEV-19834 Selectivity of an equality condition discounted twice
When discounting selectivity of ref access, don't discount the
selectivity we've already discounted for range access.

The 10.1 version of the fix. Will need to adjust condition filtering
test results in 10.4
2019-08-15 12:57:21 +03:00
Sujatha
29e560cdf3 MDEV-20348: DROP TABLE IF EXISTS killed on master but was replicated
Problem:
=======
DROP TABLE IF EXISTS was killed. The table still exists on
the master but the DDL was still logged.

Analysis:
=========
During the execution of DROP TABLE command "ha_delete_table" call is invoked
to delete the table. If the query is killed at this point, the kill command
is not handled within the code. This results in two issues.
1) The table which is not dropped also gets written into the binary log.
2) The code continues further upon receiving 'KILL QUERY'.

Fix:
===
Upon receiving the KILL command the query should stop its current execution.
Tables which were successfully dropped prior to KILL command should be
included in the binary log.
2019-08-14 22:53:16 +05:30
Vicențiu Ciorbaru
588e67956a Make sure histograms do not write uninitialized bytes to record
A histogram size that is odd in size with DOUBLE precision will leave the last
byte unwritten. When collecting histograms, this causes the last byte to
be uninitialized in the record. memset the buffer to 0 first to make
sure this does not happen.
2019-08-13 20:45:51 +03:00
Marko Mäkelä
ed4ccf34a6 Merge 10.1 into 10.2 2019-08-13 13:33:23 +03:00
Marko Mäkelä
f25e9aa4ba MDEV-20310: Make InnoDB crash tests Valgrind-friendly
Use DEBUG_SYNC to hang the execution at the interesting point,
and then kill and restart the server externally. This will work
also with Valgrind. DBUG_SUICIDE() causes Valgrind to hang,
and it could also cause uninteresting reports about memory leaks.

While we are at it, let us clean up innodb.innodb_bulk_create_index_debug
so that it will actually test the desired functionality also in future
versions (with instant ADD COLUMN and DROP COLUMN) and avoid
some unnecessary restarts.

We are adding two DEBUG_SYNC points for ALTER TABLE, because there were
none that would be executed right before ha_commit_trans().
2019-08-13 13:32:27 +03:00
Jan Lindström
5edc4ea4d9 MDEV-20324: Galera threads are not registered to performance schema
Galera threads were not registered to performance schema and
used pthread_create when mysql_thread_create should have been
used.

Added test case to verify current galera performance schema
instrumentation does work.
2019-08-13 12:52:01 +03:00
Jan Lindström
eff898f2a0 MDEV-20335: Extra trans_commit_stmt after rollback caused by incorrect fix of MDEV-14401
Fix incorrect else that should have been else if.
2019-08-13 12:37:08 +03:00
Marko Mäkelä
609ea2f37b MDEV-17614: After-merge fix
MDEV-17614 flags INSERT…ON DUPLICATE KEY UPDATE unsafe for statement-based
replication when there are multiple unique indexes. This correctly fixes
something whose attempted fix in MySQL 5.7
in mysql/mysql-server@c93b0d9a97
caused lock conflicts. That change was reverted in MySQL 5.7.26
in mysql/mysql-server@066b6fdd43
(with a substantial amount of other changes).

In MDEV-17073 we already disabled the unfortunate MySQL change when
statement-based replication was not being used. Now, thanks to MDEV-17614,
we can actually remove the change altogether.

This reverts commit 8a346f31b9 (MDEV-17073)
and mysql/mysql-server@c93b0d9a97 while
keeping the test cases.
2019-08-12 18:50:54 +03:00
Marko Mäkelä
be33124c9d Merge 10.1 into 10.2 2019-08-12 18:25:35 +03:00
Sachin
284c72eacf MDEV-17614 INSERT on dup key update is replication unsafe
Problem:-
When mysql executes INSERT ON DUPLICATE KEY INSERT, the storage engine checks
if the inserted row would generate a duplicate key error. If yes, it returns
the existing row to mysql, mysql updates it and sends it back to the storage
engine.When the table has more than one unique or primary key, this statement
is sensitive to the order in which the storage engines checks the keys.
Depending on this order, the storage engine may determine different rows
to mysql, and hence mysql can update different rows.The order that the
storage engine checks keys is not deterministic. For example, InnoDB checks
keys in an order that depends on the order in which indexes were added to
the table. The first added index is checked first. So if master and slave
have added indexes in different orders, then slave may go out of sync.

Solution:-
Make INSERT...ON DUPLICATE KEY UPDATE unsafe while using stmt or mixed format
When there is more then one unique key.
Although there is two exception.
  1. Auto Increment key is not counted because Innodb will get gap lock for
    failed Insert and concurrent insert will get a next increment value. But if
    user supplies auto inc value it can be unsafe.
  2. Count only unique keys for which insertion is performed.

So this patch also addresses the bug id #72921
2019-08-09 19:36:56 +05:30
Monty
6765cc6077 Fixed assertion Assertion `!table->pos_in_locked_tables' failed
MDEV-17717
Assertion `!table->pos_in_locked_tables' failed in tc_release_table on
flushing RocksDB table under SERIALIZABLE
MDEV-17998
Deadlock and eventual Assertion `!table->pos_in_locked_tables' failed
in tc_release_table on KILL_TIMEOUT
MDEV-19591
Assertion `!table->pos_in_locked_tables' failed in tc_release_table upon
altering table into S3 under lock.

The problem was that thd->open_tables->pos_in_locked_tables was not reset
when alter table failed to reopen a locked table.
2019-08-08 23:08:23 +03:00
Sujatha
eef7540405 MDEV-18930: Failed CREATE OR REPLACE TEMPORARY not written into binary log makes data on master and slave diverge
Problem:
=======
Failed CREATE OR REPLACE TEMPORARY TABLE statement which dropped the table but
failed at a later stage of creation of temporary table is not written to
binarylog in row based replication. This causes the slave to diverge.

Analysis:
========
CREATE OR REPLACE statements work as shown below.

CREATE OR REPLACE TABLE table_name (a int);
is basically the same as:

DROP TABLE IF EXISTS table_name;
CREATE TABLE table_name (a int);

Hence every CREATE OR REPLACE TABLE command which dropped the table should be
written to binary log, even when following CREATE TABLE part fails. In order
to achieve this, during the execution of CREATE OR REPLACE command, when a
table is dropped 'thd->log_current_statement' flag is set. When table creation
results in an error within 'mysql_create_table' code, the error handling part
looks for this flag. If it is set the failed CREATE OR REPLACE statement is
written into the binary log inspite of error. This ensure that slave doesn't
diverge from the master. In case of row based replication the error handling
code returns very early, if the table is of type temporary. This is done based
on the assumption that temporary tables are not replicated in row based
replication.

It fails to handle the cases where a temporary table was created as part of
statement based replication at an earlier stage and the binary log format was
changed to row because of an unsafe statement. In this case when a CREATE OR
REPLACE statement is executed on this temporary table it will dropped but the
query will not be written to binary log. Hence slave diverges.

Fix:
===
In error handling code check the return status of create table operation. If
it is successful and replication mode is row based and table is of type
temporary then return. Other wise proceed further to the code which checks for
thd->log_current_statement flag and does appropriate logging.
2019-08-05 14:34:31 +05:30
Sergei Petrunia
09a85692a6 Post-merge fixes for rocksdb.group_min_max test
- Fix the LooseScan code to support storage engines that return
  HA_ERR_END_OF_FILE if the index scan goes out of provided range
  bounds
- Add a DBUG_EXECUTE_IF("force_group_by",...) to allow a test to
  force a LooseScan
- Adjust rocksdb.group_min_max test not to use features not present
  in MariaDB 10.2 (e.g. optimizer_trace.  In MariaDB 10.4 it's present
  but it doesn't meet the  assumptions that the test makes about it
- Adjust the test result file:
  = MariaDB doesn't support "Enhanced Loose Scan" that FB/MySQL has
  = MariaDB has different cost calculations.
2019-08-04 17:25:17 +03:00
Oleksandr Byelkin
cf8c2a3c3b Merge branch '10.1' into 10.2 2019-07-26 07:03:39 +02:00
Oleksandr Byelkin
ae476868a5 Merge branch '5.5' into 10.1 2019-07-25 13:27:11 +02:00
Marko Mäkelä
b6ac67389d Merge 10.1 into 10.2 2019-07-25 12:14:27 +03:00
Anel Husakovic
ee555f8fc5 MDEV-19948 SHOW GRANTS return privileges individually update in 10.2 2019-07-24 23:27:10 -07:00
Sujatha
e32f29b7f3 MDEV-20091 DROP TEMPORARY table is logged despite no CREATE was logged
MDEV-5589 commit set up a policy to skip DROP TEMPORARY TABLE binary logging
in case the target table has not been "CREATEed" in binlog (no CREATE
Query-log-event was logged into the binary log).

It turns out that

1. the rule did not cover non-existing table DROPped with IF-EXISTS clause.
   The logged-create knowledge for the non-existing one does not even need
   MDEV-5589 patch, and

2. connection close disobeys it to trigger automatic DROP-IF-EXISTS
   binlogging.

Either 1 or 2 or even both is/are also responsible for unexpected binlog
records observed in MDEV-17863, actually rendering a referred
@@global.read_only irrelevant as far as the described stored procedure
definition *and* the ROW binlog-format are concerned.
2019-07-25 11:38:45 +05:30
Nisha Gopalakrishnan
2536c0b1eb BUG#28642318: POINT IN TIME RECOVERY USING MYSQLBINLOG BROKEN WITH TEMPORARY TABLE -> ERRORS
Analysis
========
Point in time recovery using mysqlbinlog containing queries
operating on temporary tables results in an error.

While writing the query log event in the binary log, the
thread id used for execution of DROP TABLE and DELETE commands
were incorrect. The thread variable 'thread_specific_used'
is used to determine whether a specific thread id is to used
while executing the statements i.e using 'SET
@@session.pseudo_thread_id'. This variable was not set
correctly for DROP TABLE query and was never set for DELETE
query. The thread id is important for temporary tables
since the tables are session specific. DROP TABLE and DELETE
queries executed using a wrong thread id resulted in errors
while applying the queries generated by mysqlbinlog utility.

Fix
===
Set the 'thread_specific_used' THD variable for DROP TABLE and
DELETE queries.

ReviewBoard: 21833
2019-07-24 18:32:24 +02:00
Gleb Shchepa
7473a71a28 Bug #29419820: MEMORY LEAK IN MY_YYOVERFLOW()
Note: this patch is for 5.6.

Detected by ASAN.

The patch fixes the cleanup of parser stack pointers.

Reviewed-by: Guilhem Bichot <guilhem.bichot@oracle.com>
2019-07-24 18:32:24 +02:00
Sergei Golubchik
8ddb7e3eb7 Bug#27167197 USING ? IN INSTALL PLUGIN QUERY ABORTS DEBUG, AND HANGS OPTIMIZED SERVER
check_valid_path() uses my_strcspn() that cannot handle invalid characters
properly. This is fixed by a big refactoring in 10.2 (MDEV-6353).

For 5.5, let's simply swap tests, because check_string_char_length()
rejects invalid characters just fine.
2019-07-24 18:32:24 +02:00
Sergei Golubchik
5e8ab9b7af Bug#27302459: EMPTY VALUE IN MYSQL.PLUGIN TABLE CAUSES SERVER TO EXIT ON STARTUP
Description:- During server startup, the server exits if
the 'mysql.plugin' system table has any rows with empty
value for the field 'name' (plugin name).
2019-07-24 18:32:24 +02:00
Georgi Kodinov
c5e9674300 Bug #27312862: ASAN: HEAP-USE-AFTER-FREE: UPDATEXML RB#21666 RB#21666
The xpath parsing function was using a local string buffer that was
deallocated when going out of scope. However references to it are
preserved in the XPATH parse tree. This was causing read-after-free.

Fixed by making the xpath buffer a local variable inside the Item
class for the relevant xpath function, thus being preserved for the
duration of the query.
2019-07-24 18:32:24 +02:00
Marko Mäkelä
e6c1e588f0 gen_lex_hash: Omit deprecated register keywords
This is follow-up to d36c107a6b
2019-07-23 15:23:27 +03:00
Sergei Golubchik
82563c5fc0 MDEV-20110 don't try to load client plugins with invalid names
reported by lixtelnis
2019-07-21 19:47:44 +02:00
Oleksandr Byelkin
f90040fd9c MDEV-19429: Wrong query result with EXISTS and LIMIT 0
Check EXISTS LIMIT before rewriting.
2019-07-19 18:00:10 +02:00
Marko Mäkelä
6962855185 Merge 10.1 into 10.2 2019-07-18 13:10:09 +03:00
Oleksandr Byelkin
cc86a0bd11 MDEV-15572: view.test, server crash with --big-tables=1
Check that table is really opened before cleanup using handler.
2019-07-18 10:01:53 +02:00
Sujatha
10ebdb7f1d MDEV-11154: Write_on_release_cache(log_event.cc) function will not write "COMMIT", if use "mysqlbinlog ... | mysql ..."
Problem:
=======
Executing command, "mysqlbinlog --read-from-remote-server --host='xx.xx.xx.xx'
--port=3306 --user=xxx --password=xxx --database=mysql --to-last-log
mysql-bin.000001 --start-position=1098699 --stop-never |mysql -uxxx -pxxx", we
found that last data read from remote couldn't commit.

Analysis:
========
The purpose of 'Write_on_release_cache' is that the contents of the Cache will
automatically be written to a dedicated result file on destruction. Flush
operation on the result file is controlled by a flag 'FLUSH_F'. Events which
require force flush upon their destruction will have to enable this
'Write_on_release_cache::FLUSH_F'. At present the 'FLUSH_F' flag is defined as
an enum as shown below.

enum flag
{
  FLUSH_F
};

Since 'FLUSH_F' is the first member without initialization it get the default
value '0'. Because of this the following flush condition never succeeds.

if (m_flags & FLUSH_F)
  fflush(m_file);

At present the file gets flushed only during my_fclose(result_file) operation.
When continuous streaming is enabled through --stop-never option it never gets
flushed and hence events are not replicated.

Fix:
===
Initialize the enum value to non zero value.
2019-07-15 13:30:10 +05:30
Jan Lindström
ec49976e38 MDEV-19746: Galera test failures because of wsrep_slave_threads identification
Problem was that tests select INFORMATION_SCHEMA.PROCESSLIST processes
from user system user and empty state. Thus, there is not clear
state for slave threads.

Changes:
- Added new status variables that store current amount of applier threads
(wsrep_applier_thread_count) and rollbacker threads
(wsrep_rollbacker_thread_count). This will make clear how many slave threads
of certain type there is.
- Added THD state "wsrep applier idle" when applier slave thread is
waiting for work. This makes finding slave/applier threads easier.
- Added force-restart option for mtr to always restart servers between tests
to avoid race on start of the test
- Added wait_condition_with_debug to wait until the passed statement returns
true, or the operation times out. If operation times out, the additional error
statement will be executed

Changes to be committed:
	new file:   mysql-test/include/force_restart.inc
	new file:   mysql-test/include/wait_condition_with_debug.inc
	modified:   mysql-test/mysql-test-run.pl
	modified:   mysql-test/suite/galera/disabled.def
	modified:   mysql-test/suite/galera/r/MW-336.result
	modified:   mysql-test/suite/galera/r/galera_kill_applier.result
	modified:   mysql-test/suite/galera/r/galera_var_slave_threads.result
	new file:   mysql-test/suite/galera/t/MW-336.cnf
	modified:   mysql-test/suite/galera/t/MW-336.test
	modified:   mysql-test/suite/galera/t/galera_kill_applier.test
	modified:   mysql-test/suite/galera/t/galera_parallel_autoinc_largetrx.test
	modified:   mysql-test/suite/galera/t/galera_parallel_autoinc_manytrx.test
	modified:   mysql-test/suite/galera/t/galera_var_slave_threads.test
	modified:   mysql-test/suite/wsrep/disabled.def
	modified:   mysql-test/suite/wsrep/r/variables.result
	modified:   mysql-test/suite/wsrep/t/variables.test
	modified:   sql/mysqld.cc
	modified:   sql/wsrep_mysqld.cc
	modified:   sql/wsrep_mysqld.h
	modified:   sql/wsrep_thd.cc
	modified:   sql/wsrep_var.cc
2019-07-15 10:17:07 +03:00
Oleksandr Byelkin
1a79a29c87 MDEV-17042: prepared statement does not return error with SQL_MODE STRICT_TRANS_TABLES.
Use for parameters value conversion functions which issue warnings.
2019-07-12 14:29:12 +02:00
Igor Babaev
8540fa83bb MDEV-19421 Basic 3-way join queries are not parsed.
The parser returned a syntax error message for the queries with join
expressions like this t1 JOIN t2 [LEFT | RIGHT] JOIN t3 ON ... ON ... when
the second operand of the outer JOIN operation with ON clause was another
join expression with ON clause. In this expression the JOIN operator is
right-associative, i.e. expression has to be parsed as the expression
t1 JOIN (t2 [LEFT | RIGHT] JOIN t3 ON ... ) ON ...
Such join expressions are hard to parse because the outer JOIN is
left-associative if there is no ON clause for the first outer JOIN operator.
The patch implements the solution when the JOIN operator is always parsed
as right-associative and builds first the right-associative tree. If it
happens that there is no corresponding ON clause for this operator the
tree is converted to left-associative.

The idea of the solution was taken from the patch by Martin Hansson
"WL#8083: Fixed the join_table rule" from MySQL-8.0 code line.
As the grammar rules related to join expressions in MySQL-8.0 and
MariaDB-5.5+ are quite different MariaDB solution could not borrow
any code from the MySQL-8.0 solution.
2019-07-11 13:39:21 -07:00
Anel Husakovic
e52fea3fe9 Fix typo open_table_def() 2019-07-11 02:59:22 -07:00
Thirunarayanan Balathandayuthapani
64900e3d7c MDEV-15641 InnoDB crash while committing table-rebuilding ALTER TABLE
Problem:
========
 There is a possibility that there can be more concurrent DMLs While the
alter table thread is waiting for upgrading to MDL_EXCLUSIVE before commit phase.
In commit phase, InnoDB acquires dict_operation_lock and it already holds MDL_EXCLUSIVE
on the table. After that, InnoDB applies the concurrent DML logs in commit phase.
This could lead to blocking of the following things:

  1) DML on the particular table (due to MDL_EXCLUSIVE on the table)
  2) InnoDB DDLs (due to dict_operation_lock)
  3) Purge thread, stats thread, the master thread (due to dict_operation_lock)

Fix:
====
Apply the concurrent DML logs in commit phase but before acquiring
dict_operation_lock in commit phase. It makes sure that (2), (3) can't be
blocked for longer time.
2019-07-10 12:43:51 +05:30
Eugene Kosov
26c389b7b7 Merge 10.1 into 10.2 2019-07-09 13:22:22 +03:00
Varun Gupta
fd08f95322 MDEV-17963: Assertion field_pos < field_count' failed in Protocol_text::store, Assertion field_handlers == 0 || field_pos < field_count'
The problem was that sp_head::MULTI_RESULTS was not set correctly for ANALYZE statement
with SELECT ... INTO variable.
This is a follow up fix for MDEV-7023
2019-07-09 10:25:22 +05:30
Aleksey Midenkov
53dd0e4f75 MDEV-16222 Assertion `0' failed in row_purge_remove_sec_if_poss_leaf on table with virtual columns and indexes
Cause
Stale thd->m_stmt_da->m_sql_errno which is from different invocation.

Fix
Reset error state before attempt to open table.
2019-07-08 20:14:51 +03:00
Eugene Kosov
838bb9fad4 fix Galera memory leak
This was caused by 7f2cfa8f47
2019-07-08 17:04:18 +03:00
Alexander Barkov
11f13bff21 MDEV-17857 Assertion `tmp != ((long long) 0x8000000000000000LL)' failed in TIME_from_longlong_datetime_packed upon SELECT with GROUP BY 2019-07-05 09:15:40 +04:00
Anel Husakovic
bf37b9fce9 MDEV-19636 Usage message for plugin activation - add FORCE_PLUS_PERMANENT 2019-07-02 12:03:39 +03:00
Vladislav Vaintroub
51c3a5c840 MDEV-19893 Do not send error packets with seqno= 0 2019-06-27 16:50:39 +02:00
Sachin
7f2cfa8f47 MDEV-8874 Replication filters configured in my.cnf are ignored if slave reset and reconfigured
Don't delete the rpl_filter on RESET SLAVE.
2019-06-27 09:54:20 +05:30
Eugene Kosov
d36c107a6b imporve clang build
cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_BUILD_TYPE=Debug

Maintainer mode makes all warnings errors. This patch fix warnings. Mostly about
deprecated `register` keyword.

Too much warnings came from Mroonga and I gave up on it.
2019-06-25 13:21:36 +03:00
Eugene Kosov
ddeeb42e0b Merge 10.1 into 10.2 2019-06-23 20:33:13 +03:00
Igor Babaev
645191aa13 MDEV-19778 Wrong Result on Left Outer Join with Subquery right on true
and WHERE filter afterwards

This patch complements the patch fixing the bug MDEV-6892. The latter
properly handled queries that used mergeable views returning constant
columns as inner tables of outer joins and whose where clause contained
predicates referring to these columns if the predicates of happened not
to be equality predicates. Otherwise the server still could return wrong
result sets for such queries. Besides the fix for MDEV-6892 prevented
some possible conversions of outer joins to inner joins for such queries.

This patch corrected the function check_simple_equality() to handle
properly conjunctive equalities of the where clause that refer to the
constant columns of mergeable views used as inner tables of an outer join.
The patch also changed the code of Item_direct_view_ref::not_null_tables().
This change allowed to take into account predicates containing references
to constant columns of mergeable views when converting outer joins into
inner joins.
2019-06-22 09:18:24 -07:00
Eugene Kosov
cf40393471 Merge 5.5 into 10.1 2019-06-20 12:26:01 +03:00
Michael Widenius
8acbf9c1f9 MDEV-19595 fixed
The test cases for the MDEV found several independent bugs
in MariaDB server and Aria:
- If a temporary table was marked as crashed, it could never
  be deleted.
- Opening of a crashed temporary table gave an error message
  but the error was never forwarded to the caller which caused
  an assert() in my_ok()
- init_read_record() did mmap of all temporary tables, which is
  probably not a good idea as this area can potentially be
  very big. Changed code to only mmap internal temporary tables.
- mmap-ed tables where not unmapped in case of repair/optimize
  which caused bad data in table and crashes if the original
  table files where replaced with new ones (as the old mmap
  was still in place). Fixed by removing the mmap in case
  of repair.
- Cleaned up usage of code that disabled mmap in Aria
2019-06-19 00:35:44 +03:00
Igor Babaev
167da05f55 MDEV-19790 Wrong result for query with outer join and IS NOT TRUE predicate
in where clause

The classes Item_func_isnottrue and Item_func_isnotfalse inherited the
implementation of the eval_not_null_tables method from the Item_func
class. As a result the not_null_tables_cache was set incorrectly for
the objects of these classes. It led to improper conversion of outer
joins to inner joins when the where clause of the processed query
contained IS NOT TRUE or IS NOT FALSE predicates. The coverted query
in many cases produced a wrong result set.
2019-06-17 14:23:10 -07:00
Michael Widenius
c8b5fa4afc MDEV-19055 Failures with temporary tables and Aria
There was two separate problems:
- Aria pagecache didn't properly handle re-reading of blocks
  that have given errors before (this triggered an assert)
- temporary tables that where opened several times where
  not properly closed in ALTER, REPAIR or OPTIMIZE table

Other things
- Added a couple of asserts that will make it easier to
  find problems like this in the future.
2019-06-17 17:50:08 +03:00
Michael Widenius
c02d6164fb MDEV-19771 REPLACE on table with virtual_field can cause crash
Fixes also MDEV-17837

Problem was that we did not ignore warnings from virtual fields when
updated virtual fields for to-be-replaced row.
2019-06-15 14:54:21 +03:00
Alexey Botchkov
65e0c9b91b MDEV-18661 loading the audit plugin causes performance regression.
Plugin fixed to not lock the LOCK_operations when not active.
Server fixed to lock the LOCK_plugin less - do it once per
thread and then only if a plugin was installed/uninstalled.
2019-06-15 01:02:55 +04:00
Sergei Golubchik
b40c2d2c51 MDEV-19633 ASAN use-after-poison in tree_insert() in main.func_gconcat
when copying the TREE, take into account that init_tree can
internally adjust tree->size_of_element
2019-06-14 12:41:06 +02:00
Oleksandr Byelkin
50653e021f Merge branch '10.1' into 10.2 2019-06-13 16:42:21 +02:00
Oleksandr Byelkin
5b65d61d93 Merge branch '5.5' into 10.1 2019-06-12 22:54:46 +02:00
Marko Mäkelä
90fec9602f Merge 10.1 into 10.2 2019-06-12 16:28:45 +03:00
Sergey Vojtovich
9d886de499 MDEV-16467 - MariaDB crashes because of "long semaphore wait"after migrating from 10.1 to 10.3
This patch fixes 10.2 issue reported in MDEV-16467 by partial backport of
c2118a0. Specifically "Remove not needed LOCK_thread_count from
thd_get_error_context_description()".
2019-06-12 13:27:43 +04:00
Marko Mäkelä
4bbd8be482 Merge 10.1 into 10.2 2019-06-12 10:30:01 +03:00
Marko Mäkelä
7a7d9904e1 MDEV-18479: Avoid COST_MULT(records, 1) 2019-06-11 12:02:26 +03:00
Igor Babaev
96ee9ea02e MDEV-18479 Another complement
This patch complements the patch that fixes bug MDEV-18479.
This patch takes care of possible overflow in JOIN::get_examined_rows().
2019-06-10 22:38:55 -07:00
Igor Babaev
6db2ebbb2a MDEV-19580 Unrelated JOINs corrupt usage of 'WHERE function() IN (subquery)'
Handling of top level conjuncts in WHERE whose used_tables() contained
RAND_TABLE_BIT in the function make_join_select() was incorrect.
As a result if such a conjunct referred to fields non of which belonged
to the last joined table  it was pushed twice. (This could be seen
for a test case from subselect.test whose output was changed after this
patch had been applied. In 10.1 when running EXPLAIN FORMAT=JSON for
the query from this test case we clearly see that one of the conjuncts
is pushed twice.) This fact by itself was not good. Besides, if such a
conjunct was pushed to a table that was the result of materialization
of a semi-join the query could return a wrong result set. In particular
we could watch it for queries with semi-join subqueries whose left parts
used stored functions without "deterministic' specifier.
2019-06-09 11:11:20 -07:00
Sergei Petrunia
5d06edfb26 MDEV-19714: JOIN::pseudo_bits_cond is not visible in EXPLAIN FORMAT=JSON
Make it visible
2019-06-08 02:28:29 +03:00
Monty
b83aff56f1 Support skip-plugin-load
- --disable-plugin-load or --skip-plugin-load will now reset plugin load
  list. Needed for future changes to mtr tests
2019-06-06 15:25:15 +03:00
Varun Gupta
7906bee67b MDEV-18015: Assertion `global_status_var.global_memory_used == 0' failed when using UDF , window functions and views
Adding destructor for Group_bound_tracker to free Cached_item_str.
The Cached_item for window functions are allocated on THD:mem_root
but the Cached_item_str has value of type string which is allocated on
the heap, so we need to call free() for it
2019-06-04 23:23:10 +05:30
Monty
76f14be10f Ensure that tests and programs can restore variables
- --default-character-set can now be disabled in mysqldump
- --skip-resolve can be be disabled in mysqld
- mysql_client_test now resets global variables it changes
- mtr couldn't handle [mysqldump] in config files (wrong regexp used)
2019-06-03 15:06:51 +03:00
Vladislav Vaintroub
2d89a70fac Fix plugin linking on Windows 2019-06-03 15:06:51 +03:00
Alexander Barkov
b003b0c934 MDEV-19675 Wrong charset is chosen when opening a pre-4.1 table 2019-06-03 12:42:36 +04:00