Commit graph

20107 commits

Author SHA1 Message Date
Daniel Black
3859273d04 MDEV-14267: correct FSF address 2018-10-30 19:45:09 +08:00
Sergei Petrunia
f8604ed9ff MDEV-17414: MyROCKS order desc limit 1 fails : Backport to 10.2
- Use the correct range bounds when doing a reverse-ordered range scan
  (this was already done for HA_READ_PREFIX_LAST_OR_PREV but not for
   HA_READ_BEFORE_KEY).
2018-10-29 13:49:44 +03:00
Marko Mäkelä
a6ffeeeaa9 MDEV-17491 micro optimize page_id_t 2018-10-29 12:05:50 +02:00
Marko Mäkelä
b3009059d0 Minor cleanup 2018-10-29 12:05:39 +02:00
Eugene Kosov
14be814380 MDEV-17491 micro optimize page_id_t
page_id_t: remove m_fold member

various places: pass page_id_t by value instead of by reference
2018-10-25 18:46:27 +03:00
Marko Mäkelä
2abf2648a6 MDEV-17536 Merge new release of InnoDB 5.7.24 to 10.2
Update the InnoDB version number to 5.7.24.
2018-10-25 17:09:54 +03:00
Marko Mäkelä
31366c6c93 MDEV-17548 Incorrect access to off-page column for indexed virtual column
row_build_index_entry_low(): ext does not contain virtual columns.

row_upd_store_v_row(): Copy virtual column values

This is based on the following fix in MySQL 5.7.24:

commit 4ec2158bec73f1582501c4b3e3de250fed9edc9a
Author: Sachin Agarwal <sachin.z.agarwal@oracle.com>
Date:   Fri Aug 24 14:44:13 2018 +0530

    Bug  INNODB CRASH/CORRUPTION WITH TEXT PREFIX INDEXES

    Problem:
    There are two problems:
      1. If there is one secondary index on extenally
    stored column and another seconday index on virtual column (whose
    base column is not externally stored). then while updating seconday
    index on vitrual column, virtual column data is replaced by
    externally stoared column.
      2. In row update operation, node->row contains
    shallow copy of virtual data fields. While building an update vector
    containing all the fields to be modified, compute virtual column.
    which may causes change in virtual data fields in node->row.

    In both the above cases, while updating seconday index on virtual
    column, couldn't find the row and hit an explicite assert inside
    ROW_NOT_FOUND.

    Fix:
    1. Added check if column is virtual then its ext flag should be ZERO
    and virtual column data will not be replaced by offset column data.
    2. Deep copy of virtual data fields for node->row.

    RB: 
    Reviewed by : Jimmy.Yang@oracle.com
2018-10-25 17:08:36 +03:00
Marko Mäkelä
e548d92b23 MDEV-17546 SPATIAL INDEX should not be allowed for FOREIGN KEY
dict_foreign_qualify_index(): Reject both FULLTEXT and SPATIAL
indexes. Remove these checks from the callers.

It is unclear whether this bug affected MariaDB Server.
FOREIGN KEY constraints on geometry column types could have
been rejected due to column type restrictions.
2018-10-25 16:11:52 +03:00
Marko Mäkelä
7e9728a913 MDEV-17545 Predicate lock for SPATIAL INDEX should lock non-matching record
rtr_pcur_getnext_from_path(): Remove a bogus condition.
The predicate lock should be acquired also on no match,
to ensure that the locking read will be repeatable.

This is based on the following fix in MySQL 5.7.24:

commit 365111c590082984dbae42e1d1da28ac3f7fb5bd
Author: Jimmy Yang <jimmy.yang@oracle.com>
Date:   Wed Jun 6 16:23:00 2018 -0700

    Fix Bug 27577612 - CONCURRENT SERIALIZABLE TRANSACTIONS CAN INSERT INTO
    AN AREA SELECTED FOR UPDATE
    Backport fix to mysql-5.7

    Reviewed-by: Allen Lai <allen.lai@oracle.com>

No test case is added, because the MySQL 5.7 test case would pass
even when the fix is not present. We would need a test case that
only causes a locking conflict on the spatial index.
2018-10-25 15:18:09 +03:00
Marko Mäkelä
a21e01a53d MDEV-17541 KILL QUERY during lock wait in FOREIGN KEY check causes hang
row_ins_check_foreign_constraint(): Do not overwrite hard errors
with the soft error DB_LOCK_WAIT. This prevents an infinite
wait loop when DB_INTERRUPTED was returned. For DB_LOCK_WAIT,
row_insert_for_mysql() would keep invoking row_ins_step() and the
transaction would remain active until the server shutdown is initiated.
2018-10-25 10:35:46 +03:00
Marko Mäkelä
ab1ce2204e MDEV-17466: Remove the debug assertion
This reverts commit 2d4075e1d9
where the debug assertion was added. There seems to be a potential
problem in the purge of indexes that depend on virtual columns.

Ultimately, we should change the InnoDB undo log format so that
all actual secondary index keys are stored there, also for
virtual or spatial indexes. In that way, purge and rollback would
be more straightforward.
2018-10-19 09:31:33 +03:00
Marko Mäkelä
abbf169f52 Remove unused TIMETPF 2018-10-19 09:28:21 +03:00
Marko Mäkelä
64d26ec52a Fix the Windows build
On Windows, a mismatch between TIMETPF ("%ld") and time_t would be reported.
Use "%ld" and long, like the code used to be.
2018-10-18 12:15:07 +03:00
Marko Mäkelä
56de291978 Fix a merge error in commit 28f08d3753 2018-10-18 10:10:11 +03:00
Marko Mäkelä
853a0a4368 MDEV-13564: Set innodb_safe_truncate=ON by default
The setting innodb_safe_truncate=ON reduces compatibility with older
versions of MariaDB and backup tools in two ways.

First, we will be writing TRX_UNDO_RENAME_TABLE records, which older
versions do not know about. These records could be misinterpreted if
a DDL transaction was recovered and would be rolled back.
Such rollback is only possible if the server was killed while
an incomplete DDL transaction was persisted. On transaction completion,
the insert_undo log pages would only be repurposed for new undo log
allocations, and their contents would not matter. So, older versions
will not have a problem with innodb_safe_truncate=ON if the server was
shut down cleanly.

Second, to prevent such recovery failure, innodb_safe_truncate=ON will
cause a modification of the redo log format identifier, which will
prevent older versions from starting up after a crash. MariaDB Server
versions older than 10.2.13 will refuse to start up altogether, even
after clean shutdown.

A server restart with innodb_safe_truncate=OFF will restore compatibility
with older server and backup versions.
2018-10-17 17:35:38 +03:00
Marko Mäkelä
2d4075e1d9 MDEV-17466 Virtual column value not available during purge
row_build_index_entry_low(): Assert that when the value of a
virtual column is not available, this can only happen when
the index creation was completed but not committed yet.

This change is not fixing any bug, making a debug assertion
stricter, so that bugs can be caught in the future.

Ultimately, we should change the InnoDB undo log format so that
all actual secondary index keys are stored there, also for
virtual or spatial indexes. In that way, purge and rollback would
be more straightforward.
2018-10-16 10:42:30 +03:00
Elena Stepanova
b715a0fe45 Disabled storage_engine test for RocksDB due to unstable execution plan
The failure caused by MDEV-16387
2018-10-13 19:14:31 +03:00
Marko Mäkelä
81a5b6ccd5 MDEV-17433 Allow InnoDB start up with empty ib_logfile0 from mariabackup --prepare
A prepared backup from Mariabackup does not really need to contain any
redo log file, because all log will have been applied to the data files.

When the user copies a prepared backup to a data directory (overwriting
existing files), it could happen that the data directory already contained
redo log files from the past. mariabackup --copy-back) would delete the
old redo log files, but a user’s own copying script might not do that.
To prevent corruption caused by mixing an old redo log file with data
files from a backup, starting with MDEV-13311, Mariabackup would create
a zero-length ib_logfile0 that would prevent startup.

Actually, there is no need to prevent InnoDB from starting up when a
single zero-length file ib_logfile0 is present. Only if there exist
multiple data files of different lengths, then we should refuse to
start up due to inconsistency. A single zero-length ib_logfile0 should
be treated as if the log files were missing: create new log files
according to the configuration.

open_log_file(): Remove. There is no need to open the log files
at this point, because os_file_get_status() already determined
the size of the file.

innobase_start_or_create_for_mysql(): Move the creation of new
log files a little later, not when finding out that the first log
file does not exist, but after finding out that it does not exist
or it exists as a zero-length file.
2018-10-11 23:00:48 +03:00
Marko Mäkelä
b8944e8972 Fix a sign mismatch 2018-10-11 22:47:42 +03:00
Marko Mäkelä
6319c0b541 MDEV-13564: Replace innodb_unsafe_truncate with innodb_safe_truncate
Rename the 10.2-specific configuration option innodb_unsafe_truncate
to innodb_safe_truncate, and invert its value.

The default (for now) is innodb_safe_truncate=OFF, to avoid
disrupting users with an undo and redo log format change within
a Generally Available (GA) release series.
2018-10-11 15:10:13 +03:00
Marko Mäkelä
3448ceb02a MDEV-13564: Implement innodb_unsafe_truncate=ON for compatibility
While MariaDB Server 10.2 is not really guaranteed to be compatible
with Percona XtraBackup 2.4 (for example, the MySQL 5.7 undo log format
change that could be present in XtraBackup, but was reverted from
MariaDB in MDEV-12289), we do not want to disrupt users who have
deployed xtrabackup and MariaDB Server 10.2 in their environments.

With this change, MariaDB 10.2 will continue to use the backup-unsafe
TRUNCATE TABLE code, so that neither the undo log nor the redo log
formats will change in an incompatible way.

Undo tablespace truncation will keep using the redo log only. Recovery
or backup with old code will fail to shrink the undo tablespace files,
but the contents will be recovered just fine.

In the MariaDB Server 10.2 series only, we introduce the configuration
parameter innodb_unsafe_truncate and make it ON by default. To allow
MariaDB Backup (mariabackup) to work properly with TRUNCATE TABLE
operations, use loose_innodb_unsafe_truncate=OFF.

MariaDB Server 10.3.10 and later releases will always use the
backup-safe TRUNCATE TABLE, and this parameter will not be
added there.

recv_recovery_rollback_active(): Skip row_mysql_drop_garbage_tables()
unless innodb_unsafe_truncate=OFF. It is too unsafe to drop orphan
tables if RENAME operations are not transactional within InnoDB.

LOG_HEADER_FORMAT_10_3: Replaces LOG_HEADER_FORMAT_CURRENT.

log_init(), log_group_file_header_flush(),
srv_prepare_to_delete_redo_log_files(),
innobase_start_or_create_for_mysql(): Choose the redo log format
and subformat based on the value of innodb_unsafe_truncate.
2018-10-11 08:17:04 +03:00
Marko Mäkelä
07815d9555 Merge 10.1 into 10.2 2018-10-11 08:16:08 +03:00
Marko Mäkelä
940f0c78a4 MDEV-11487: Make row_ins_index_entry_set_vals() static 2018-10-11 08:14:56 +03:00
Sergei Petrunia
8d116d1686 MDEV-17181: rocksdb.allow_to_start_after_corruption fails on current 10.2
The test needs to be run with rocksdb_flush_log_at_trx_commit=1, otherwise
the changes do not survive a crash.
2018-10-10 14:39:57 +03:00
Sergei Petrunia
8b371e4b13 MDEV-16577: rocksdb.issue255 fails in buildbot
Make the testcase stable
2018-10-09 17:01:49 +03:00
Thirunarayanan Balathandayuthapani
e9d9ca8c44 MDEV-16980 Wrongly set tablename len while opening the
table for purge thread

Problem:
=======
	Purge tries to fetch mdl lock for the whole table even though it tries
to open one of the partition. But table name length was wrongly set to indicate
the partition name too.

Solution:
========
- Table name length should identify the table name only not the partition name.
2018-10-08 21:40:18 +05:30
Vladislav Vaintroub
acca321af3 CMake, Windows - reduce amount of noisy, irrelevant MESSAGE()s 2018-10-05 16:48:51 +01:00
Michael Widenius
1e06daea7c Remove not used variable 2018-10-05 14:25:40 +03:00
Thirunarayanan Balathandayuthapani
2af67150cf MDEV-17289 Multi-pass recovery fails to apply some redo log records
This is a regression caused by
commit 73af8af094
(MDEV-15325 Incomplete validation of missing tablespace during recovery).

If the recv_sys->addr_hash hash table ran out of memory, we would
have to do crash recovery in multiple passes. If some tablespaces were
missing, after the MDEV-15325 fix we would rescan the remaining redo log.
But, we could incorrectly reset the "rescan" flag. Because of this, we
would fail to apply some of the oldest redo log records to the data files.
(The recv_sys->addr_hash would only contain records from the latest
redo log scan batch.)

Fix:

After checking for missing tablespaces, reset the flag rescan=true,
so that all redo log records will be re-read and applied.
2018-10-05 16:44:51 +05:30
Eugene Kosov
15803fce92 MDEV-17313 Data race in ib_counter_t
ib_counter_t: make all reads/writes to m_counter relaxed atomical
2018-10-02 13:34:11 +03:00
Thirunarayanan Balathandayuthapani
b4e841648c MDEV-17215 Assertion `rw_lock_own(dict_operation_lock, RW_LOCK_S)
|| node->vcol_info.is_used()' failed

- Purge thread can acquire mdl lock while initializing the mysql template.
Set the vcol_info information before acquiring mdl lock.

- Purge thread doesn't need to use the virtual column info even though it is
requested. In that case, reset the virtual column info.
2018-10-01 13:23:33 +05:30
Eugene Kosov
0f709912fb MDEV-17306 rw_lock_x_lock_wait_func() double increments rw_x_spin_round_count
rw_lock_x_lock_wait_func(): remove duplicated logic added in incorrect merge

Affected counters are affected by InnoDB monitor. But they aren't stable and thus
can not be realiably tested.
2018-09-27 17:57:27 +03:00
Marko Mäkelä
f77071e0e8 Remove unused code
Some global function definitions were orphaned by a bug fix in
MySQL 5.7.14 more than 2 years ago.
Remove them.
5ed18d823c
2018-09-26 17:04:38 +03:00
Marko Mäkelä
304857764b MDEV-13564 Mariabackup does not work with TRUNCATE
Implement undo tablespace truncation via normal redo logging.

Implement TRUNCATE TABLE as a combination of RENAME to #sql-ib name,
CREATE, and DROP.
2018-09-25 17:29:43 +03:00
Sergei Golubchik
339edd462f fixed auto-merge gone bad 2018-09-24 20:37:22 +02:00
Sergei Golubchik
5ae8fce50b Merge branch '10.1' into 10.2 2018-09-24 11:46:08 +02:00
Sergei Golubchik
76098f45b8 RocksDB: workaround a compiler error on ppc64le
storage/rocksdb/rdb_datadic.cc: In member function 'int myrocks::Rdb_key_def::unpack_integer(myrocks::Rdb_field_packing*, Field*, uchar*, myrocks::Rdb_string_reader*, myrocks::Rdb_string_reader*) const'
storage/rocksdb/rdb_datadic.cc:1781:1: internal compiler error: Segmentation fault
 }

on ppc64le, ubuntu bionic gcc 7.3.0 and debian stretch gcc 6.3.0

The error happens with -ftree-loop-vectorize when trying to vectorize
a particular loop (see Rdb_key_def::unpack_integer())

Compiler gets confused by __attribute__((optimize("O0")) that comes from
ha_rocksdb_proto.h. The intention of this __attribute__ was to prevent
function from being inlined (see ha_rocksdb.cc). Let's use a more
specific attribute that prevents inlining but does not confuse
loop vectorizer.
2018-09-23 19:17:56 +02:00
Sergei Golubchik
1fc5a6f30c Merge branch '10.0' into 10.1 2018-09-23 12:58:11 +02:00
Sergei Petrunia
2b45eb77f7 MDEV-17261: sysbench oltp read only too slow for MyRocks
An error in "group commit with MariaDB's binlog" code: we would flush
the WAL even when the transaction did not do any writes (and so the logic
in myrocks::Rdb_transaction::commit caused it to rollback).
2018-09-23 13:41:08 +03:00
Sergei Golubchik
1144acbcbd tokudb: create and destroy TOKUDB_SHARE::_open_tables_mutex dynamically
to guarantee that it's destroyed when plugin deinit is called, not after
2018-09-22 20:18:17 +02:00
Sergei Golubchik
3a9276bad3 sanitize tokudb locking macros 2018-09-22 15:19:40 +02:00
Sergei Golubchik
1ebec10375 InnoDB: fix compilation with -DDBUG_OFF 2018-09-21 13:31:37 +02:00
Marko Mäkelä
4fa9eaf454 Remove unused ha_innobase::lock
In MySQL 5.7, a follow-up to WL#6671 removed the unused
fields ha_innobase::lock and INNOBASE_SHARE::lock, but
MariaDB did not remove them, even though a counterpart of
WL#6671 itself was implemented as MDEV-7660 in
commit d665e79c5b.

INNOBASE_SHARE was removed in MDEV-16557. Thus, all that
needs to be removed is the unused member ha_innobase::lock
and related code.

Thanks to Monty (and Valgrind) for noticing that
ha_innobase::lock was uninitialized.
2018-09-17 08:55:56 +03:00
Oleksandr Byelkin
28f08d3753 Merge branch '10.1' into 10.2 2018-09-14 08:47:22 +02:00
Jacob Mathew
6c47c1c456 MDEV-16912: Spider Order By column[datatime] limit 5 returns 3 rows
The problem occurs in 10.2 and earlier releases of MariaDB Server because the
Partition Engine was not pushing the engine conditions to the underlying
storage engine of each partition.  This caused Spider to return the first 5
rows in the table with the data provided by the customer.  2 of the 5 rows
did not qualify the WHERE clause, so they were removed from the result set by
the server.

To fix the problem, I have back-ported support for engine condition pushdown
in the Partition Engine from MariaDB Server 10.3.

Author:
  Jacob Mathew.

Reviewer:
  Kentoku Shiba.

Cherry-Picked:
  Commit eb2ca3d on branch bb-10.2-MDEV-16912
2018-09-13 13:27:03 -07:00
Jacob Mathew
3866589308 MDEV-16912: Spider Order By column[datatime] limit 5 returns 3 rows
The problem occurs in 10.2 and earlier releases of MariaDB Server because the
Partition Engine was not pushing the engine conditions to the underlying
storage engine of each partition.  This caused Spider to return the first 5
rows in the table with the data provided by the customer.  2 of the 5 rows
did not qualify the WHERE clause, so they were removed from the result set by
the server.

To fix the problem, I have back-ported support for engine condition pushdown
in the Partition Engine from MariaDB Server 10.3.

Author:
  Jacob Mathew.

Reviewer:
  Kentoku Shiba.

Merged:
  Commit eb2ca3d on branch bb-10.2-MDEV-16912
2018-09-13 11:41:55 -07:00
Sergei Petrunia
d85a7220dc MDEV-17188: rocksdb.2pc_group_commit fails intermittently in BB
When counter increment is not within the expected range, print the number
instead of just FAIL.

This doesnt solve the bug but will help with the diagnostics.
2018-09-13 15:00:13 +03:00
Oleksandr Byelkin
2b46dca5d7 Merge remote-tracking branch 'connect/10.2' into 10.2 2018-09-13 10:09:28 +02:00
Jacob Mathew
eb2ca3d445 MDEV-16912: Spider Order By column[datatime] limit 5 returns 3 rows
The problem occurs in 10.2 and earlier releases of MariaDB Server because the
Partition Engine was not pushing the engine conditions to the underlying
storage engine of each partition.  This caused Spider to return the first 5
rows in the table with the data provided by the customer.  2 of the 5 rows
did not qualify the WHERE clause, so they were removed from the result set by
the server.

To fix the problem, I have back-ported support for engine condition pushdown
in the Partition Engine from MariaDB Server 10.3.

Author:
  Jacob Mathew.

Reviewer:
  Kentoku Shiba.
2018-09-11 16:29:44 -07:00
Marko Mäkelä
fc34e4c067 MDEV-17161 TRUNCATE TABLE fails after upgrade from 10.1
With the TRUNCATE by rename, create, drop (MDEV-13564),
old tables with invalid ROW_FORMAT attribute could not be
truncated. Introduce a sloppy mode for allowing the TRUNCATE.

create_table_info_t::prepare_create_table(): Add the parameter
strict=true.

ha_innobase::create(): Pass strict=false if trx!=NULL
(the create is part of TRUNCATE).
2018-09-10 16:10:26 +03:00