Commit graph

21392 commits

Author SHA1 Message Date
Igor Babaev
7d9f45e072 Merge branch '10.4' into bb-10.4-mdev17096 2019-02-13 14:59:34 -08:00
Marko Mäkelä
a081a998a6 MDEV-15563: Fix cmake -DPLUGIN_PERFSCHEMA=NO
Commit 22feb179ae broke the build
with performance_schema disabled.

dict_col_t::same_charset(): An auxiliary function to
dict_col_t::same_format(). Determine if two non-binary string
columns have the same character set.
2019-02-13 22:11:40 +02:00
Igor Babaev
62fad4e8e9 MDEV-17096 Pushdown of simple derived tables to storage engines
Fixing failures of federated test on 32-bit platforms
2019-02-13 08:55:38 -08:00
Marko Mäkelä
22feb179ae MDEV-15563: Instant ROW_FORMAT=REDUNDANT column extension
This was developed by Aleksey Midenkov based on my design.

In the original InnoDB storage format (that was retroactively named
ROW_FORMAT=REDUNDANT in MySQL 5.0.3), the length of each index field
is stored explicitly.

Because of this, we can and now will allow instant conversion from
VARCHAR to CHAR or VARBINARY to BINARY of equal or greater size,
as well as instant conversion of TINYINT to SMALLINT to MEDIUMINT
to INT to BIGINT (while not changing between signed and unsigned).

Theoretically, we could allow changing from an unsigned integer to
a bigger unsigned integer, as well as changing CHAR to VARCHAR, but
that would require additional metadata and conversions whenever
reading old records.

Field_str::is_equal(), Field_varstring::is_equal(), Field_num::is_equal():
Return the new result IS_EQUAL_PACK_LENGTH_EXT if the table advertises
HA_EXTENDED_TYPES_CONVERSION capability and we are considering the
above-mentioned conversions.

ALTER_COLUMN_EQUAL_PACK_LENGTH_EXT: A new ALTER TABLE flag, similar
to ALTER_COLUMN_EQUAL_PACK_LENGTH but requiring conversions when
reading the data. The Field::is_equal() result IS_EQUAL_PACK_LENGTH_EXT
will map to this flag.

dtype_get_fixed_size_low(): For BINARY, CHAR and integer columns
in ROW_FORMAT=REDUNDANT, return 0 (variable length) from now on.

dtype_get_sql_null_size(): Keep returning the current size for
BINARY, CHAR and integer columns, so that in ROW_FORMAT=REDUNDANT
it will remain possible to update in place between NULL and NOT NULL
values.

btr_index_rec_validate(): Relax a CHECK TABLE length check for
ROW_FORMAT=REDUNDANT tables.

btr_cur_instant_init_low(): No longer trust fixed_len
for ROW_FORMAT=REDUNDANT tables.

We cannot rely on fixed_len anymore because the record can have shorter
length from before instant extension. Note that importing such tablespace
into earlier MariaDB versions produces ER_TABLE_SCHEMA_MISMATCH when
using a .cfg file.
2019-02-13 17:39:05 +02:00
Marko Mäkelä
0ae3ea7919 MDEV-15563: Instant VARCHAR extension for ROW_FORMAT=REDUNDANT
In the original InnoDB storage format (which was retroactively named
ROW_FORMAT=REDUNDANT in MySQL 5.0.3), the length of each index field
is stored explicitly. Thus, we can and from now on will allow arbitrary
extension of VARBINARY and VARCHAR columns when the table is in
ROW_FORMAT=REDUNDANT.

ha_innobase::open(): Advertise a new HA_EXTENDED_TYPES_CONVERSION
capability for ROW_FORMAT=REDUNDANT tables.

Field_varstring::is_equal(): If the HA_EXTENDED_TYPES_CONVERSION
capability is advertised for the table, return IS_EQUAL_PACK_LENGTH
for any length extension.
2019-02-13 16:42:03 +02:00
Igor Babaev
953ca199fb Merge branch '10.4' into bb-10.4-mdev17096 2019-02-12 23:19:43 -08:00
Igor Babaev
27c3abde30 MDEV-17096 Pushdown of simple derived tables to storage engines
MDEV-17631 select_handler for a full query pushdown

Added comments and file headers for files introduced in these tasks.
2019-02-12 22:56:24 -08:00
Igor Babaev
58b950737c MDEV-17096 Pushdown of simple derived tables to storage engines
Added plugin system variable federated_pushdown.
2019-02-12 13:11:32 -08:00
Marko Mäkelä
9f56dd7382 Merge 10.3 into 10.4 2019-02-11 17:55:25 +02:00
Marko Mäkelä
4e7ee166a9 MDEV-18295 IMPORT TABLESPACE fails with instant-altered tables
When importing a tablespace, we must initialize dummy DEFAULT NULL
values for any instantly added columns in order to avoid a debug
assertion failure when PageConverter::update_records() invokes
rec_get_offsets(). Finally, when the operation completes, we must
evict and reload the table definition, so that the correct
default values for instantly added columns will be loaded.

ha_innobase::discard_or_import_tablespace(): On successful
IMPORT TABLESPACE, evict and reload the table definition,
so that btr_cur_instant_init() will load the correct metadata.

PageConverter::update_index_page(): Fill in dummy DEFAULT NULL values
for instantly added columns. These will be replaced upon the
completion of the operation by evicting and reloading the metadata.

row_discard_tablespace(): Invoke dict_table_t::remove_instant().
After DISCARD TABLESPACE, the table is no longer in "instant ALTER"
format, because there is no data file attached.
2019-02-11 14:42:48 +02:00
Igor Babaev
d11be23933 MDEV-17096 Pushdown of simple derived tables to storage engines
Resolved the problem of forming a proper query string for FEDERATEDX.
Added test cases.

Cleanup of extra spaces.
2019-02-09 22:54:26 -08:00
Marko Mäkelä
3c305d3f19 MDEV-15528: Introduce MLOG_INIT_FREE_PAGE
This is only a placeholder that allows an implementation later
during the development of MariaDB, so that downgrade to an earlier
version (with this code) will be possible.
We want to be able to zero out freed pages to reduce write amplification,
and to scrub old data. Zeroing out the pages is optional, not mandatory
for correctness. After all, the MLOG_INIT_FREE_PAGE record can only be
emitted for pages that are marked free in the allocation bitmap page.
2019-02-08 19:16:32 +02:00
Marko Mäkelä
ea25bdc135 Do not write MLOG_IBUF_BITMAP_INIT
Use ibuf_bitmap_page_init() only during recovery.

fsp_fill_free_list(): Initialize the FIL_PAGE_TYPE using MLOG_2BYTES.
The page contents will already have been zeroed out by
MLOG_INIT_FILE_PAGE2.

ibuf_bitmap_init_apply(): Replaces ibuf_parse_bitmap_init().
2019-02-08 19:10:33 +02:00
Marko Mäkelä
b8e867e869 MDEV-18219 Assertion index->n_core_null_bytes <= ... after DROP COLUMN
rec_init_offsets(): Relax a debug assertion that is too strict
during the execution of btr_cur_instant_init_low().
2019-02-08 18:46:03 +02:00
Teemu Ollakka
58de2bf30d MDEV-18481 Use mlog_memset() to clear wsrep checkpoint 2019-02-08 14:17:23 +00:00
Marko Mäkelä
f4f8dd69aa MDEV-18493: Correct a bogus assertion 2019-02-07 16:25:18 +02:00
Marko Mäkelä
0a1c3477bf MDEV-18493 Remove page_size_t
MySQL 5.7 introduced the class page_size_t and increased the size of
buffer pool page descriptors by introducing this object to them.

Maybe the intention of this exercise was to prepare for a future
where the buffer pool could accommodate multiple page sizes.
But that future never arrived, not even in MySQL 8.0. It is much
easier to manage a pool of a single page size, and typically all
storage devices of an InnoDB instance benefit from using the same
page size.

Let us remove page_size_t from MariaDB Server. This will make it
easier to remove support for ROW_FORMAT=COMPRESSED (or make it a
compile-time option) in the future, just by removing various
occurrences of zip_size.
2019-02-07 12:21:35 +02:00
Eugene Kosov
10dac4293f MDEV-18444 ROW_FORMAT=COMPRESSED unnecessarily requires NOCOPY for INSTANT operation
instant_alter_column_possible(): allow non-rebuilding operations for
ROW_FORMAT=COMPRESSED
2019-02-07 12:20:53 +02:00
Igor Babaev
3f9040085a Merge branch '10.4' into bb-10.4-mdev17096 2019-02-06 18:01:29 -08:00
Igor Babaev
16327fc2e7 MDEV-17096 Pushdown of simple derived tables to storage engines
MDEV-17631 select_handler for a full query pushdown

Interfaces + Proof of Concept for federatedx with test cases.

The interfaces have been developed for integration of ColumnStore engine.
2019-02-06 17:02:44 -08:00
Monty
d0799a0479 Removed compiler warnings from tokudb
- Backport from 10.4
2019-02-06 22:18:20 +02:00
Marko Mäkelä
e80bcd7f64 Merge 10.3 into 10.4 2019-02-05 12:48:02 +02:00
Marko Mäkelä
ab2458c61f Merge 10.2 into 10.3 2019-02-04 15:12:14 +02:00
Kentoku
5b996782be MDEV-16279 Spider crashes on CHECKSUM TABLE with spider_quick_mode=3
add tests
2019-02-03 15:58:45 +05:30
Kentoku
540e82d1b4 MDEV-16787 optimistic parallel replication fails on spider
Add a system variable spider_slave_trx_isolation.
- spider_slave_trx_isolation
  The transaction isolation level when Spider table is used by slave SQL thread.
  -1 : OFF
   0 : READ UNCOMMITTED
   1 : READ COMMITTED
   2 : REPEATABLE READ
   3 : SERIALIZABLE
  The default value is -1

Miscellaneous Spider typos
2019-02-03 15:58:45 +05:30
Kentoku
41e60e7fe8 MDEV-16520 Out-Of-Memory running big aggregate query on Spider Engine
Change default value of the followings
quick_mode 0 -> 3
quick_page_size 100 -> 1024

Add the following parameter for limiting result page size by byte
- quick_page_byte(qpb)
  Number of bytes in a page when acquisition one by one.
  When quick_mode is 1 or 2, Spider stores at least 1 record even if
  quick_page_byte is smaller than 1 record. When quick_mode is 3,
  quick_page_byte is used for judging using temporary table.
  That is given to priority when server parameter spider_quick_page_byte
  is set.
  The default value is 10485760

Fix "out of sync" issue at using quick_mode = 1 or 2
2019-02-03 15:58:45 +05:30
Kentoku
b27284db4a MDEV-16279 Spider crashes on CHECKSUM TABLE with spider_quick_mode=3
The fields of the temporary table were not created in create_tmp_table function. Because item->const_item() was true. But the temporary tables that is created by Spider are always used all columns. So Spider should call create_tmp_table function with TMP_TABLE_ALL_COLUMNS flag.
2019-02-03 15:58:45 +05:30
Kentoku
4a28a79e48 Update Spider to version 3.3.14. Add direct left outer join/right outer join/inner join feature 2019-02-03 15:58:45 +05:30
Thirunarayanan Balathandayuthapani
7c7161a1bd MDEV-18194 Incremental prepare tries to access page which is out of tablespace bounds
Problem:
=======
Mariabackup incremental prepare creates new tablespace when it encounter
new tablespace. It sets the intial size as FIL_IBD_FILE_INITIAL_SIZE (4).
But while applying redo log, it tries to access 5th page and then
it leads to out of tablespace error.

Fix:
===
While parsing the redo log record, track FSP_SIZE in recv_spaces for the
respective space id. Assign the recv_size for the tablespace when it
is loaded. Extend the tablespace depends on recv_size while applying
the redo log record.
2019-02-01 09:15:53 +02:00
Thirunarayanan Balathandayuthapani
f669cecbe3 MDEV-18415 mariabackup.mdev-14447 test case fails with Table 'test.t' doesn't exist in engine
- Added retry logic if validation of first page fails with checksum
mismatch.
2019-02-01 08:53:50 +02:00
Kentoku
4e599c74bb MDEV-18313 Supports 'wrapper mariadb' for connection information 2019-01-31 08:51:34 +01:00
Kentoku
470c1b8d56 Fix an error at using spider_direct_sql with temporary table 2019-01-31 08:51:34 +01:00
Kentoku
ea347fcb28 remove unnecessary drop database/table from slave_trx_isolation.test
add simplified slave_trx_isolation.test
2019-01-31 08:51:34 +01:00
Kentoku
be556f817b remove unnecessary drop database/table from quick_mode.test
add simplified quick_mode.test
2019-01-31 08:51:34 +01:00
Kentoku
6d80d35d91 MDEV-16787 optimistic parallel replication fails on spider
Add a system variable spider_slave_trx_isolation.
- spider_slave_trx_isolation
  The transaction isolation level when Spider table is used by slave SQL thread.
  -1 : OFF
   0 : READ UNCOMMITTED
   1 : READ COMMITTED
   2 : REPEATABLE READ
   3 : SERIALIZABLE
  The default value is -1

Miscellaneous Spider typos
2019-01-31 08:51:34 +01:00
Kentoku
bef6b197fc MDEV-16520 Out-Of-Memory running big aggregate query on Spider Engine
Change default value of the followings
quick_mode 0 -> 3
quick_page_size 100 -> 1024

Add the following parameter for limiting result page size by byte
- quick_page_byte(qpb)
  Number of bytes in a page when acquisition one by one.
  When quick_mode is 1 or 2, Spider stores at least 1 record even if
  quick_page_byte is smaller than 1 record. When quick_mode is 3,
  quick_page_byte is used for judging using temporary table.
  That is given to priority when server parameter spider_quick_page_byte
  is set.
  The default value is 10485760

Fix "out of sync" issue at using quick_mode = 1 or 2
2019-01-31 08:51:34 +01:00
Kentoku
3cb7c5f27b MDEV-16279 Spider crashes on CHECKSUM TABLE with spider_quick_mode=3
The fields of the temporary table were not created in create_tmp_table function. Because item->const_item() was true. But the temporary tables that is created by Spider are always used all columns. So Spider should call create_tmp_table function with TMP_TABLE_ALL_COLUMNS flag.
2019-01-31 08:51:34 +01:00
Kentoku
6caf9ec425 Update Spider to version 3.3.14. Add direct left outer join/right outer join/inner join feature 2019-01-31 08:51:34 +01:00
Thirunarayanan Balathandayuthapani
b8aef87221 MDEV-16849 Extending indexed VARCHAR column should be instantaneous
Analysis:
========
Increasing the length of the indexed varchar column is not an instant operation for
innodb.

Fix:
===
- Introduce the new handler flag 'Alter_inplace_info::ALTER_COLUMN_INDEX_LENGTH' to
indicate the index length differs due to change of column length changes.

- InnoDB makes the ALTER_COLUMN_INDEX_LENGTH flag as instant operation.

This is a port of Mysql fix.

    commit 913071c0b16cc03e703308250d795bc381627e37
    Author: Nisha Gopalakrishnan <nisha.gopalakrishnan@oracle.com>
    Date:   Wed May 30 14:54:46 2018 +0530

        BUG#26848813: INDEXED COLUMN CAN'T BE CHANGED FROM VARCHAR(15)
                      TO VARCHAR(40) INSTANTANEOUSLY
2019-01-30 15:33:32 +05:30
Sergey Vojtovich
8553525931 MDEV-18400 - Move shutdown handling to main thread
Signal handler is now respoinsible for setting abort_loop and breaking
poll() in main thread. The rest is handled by main thread itself.

Removed redundant LOCK_error_log init/destroy wrappers.
Removed redundant unireg_end(): it is trivial and it has only one caller.
Removed unused ready_to_exit from PFS.
Removed kill_in_progress: duplicates abort_loop.
Removed shutdown_in_progress: duplicates abort_loop.
Removed ready_to_exit: was used to make sure main thread waits for
cleanups, which are now done by main thread itself.
Removed SIGNALS_DONT_BREAK_READ, MAYBE_BROKEN_SYSCALL,
kill_broken_server: never defined/used.
Make clean_up() static.
2019-01-29 11:56:35 +04:00
Marko Mäkelä
36be0a5aef MDEV-18399 Recognize the deprecated parameters innodb_file_format, innodb_large_prefix
The parameters innodb_file_format and innodb_large_prefix were overridden
in the Debian-distributed configuration files, because the default values
of these parameters between MariaDB 5.5 and MariaDB 10.2
did not make any sense.

To allow a more seamless upgrade from MariaDB 10.1 to later versions,
let InnoDB recognize the parameters innodb_file_format and
innodb_large_prefix and issue deprecation warnings for them if they
are specified. A deprecation period of only one major release
(one year between the MariaDB 10.2 and 10.3 releases) is insufficient
for these widely used parameters.
2019-01-28 17:58:14 +02:00
Jan Lindström
97930df13c
Merge pull request from codership/10.2-MDEV-15740
MDEV-15740 Fixes to Galera transaction recovery
2019-01-28 12:01:35 +02:00
Sergei Golubchik
9b76e2843b Merge branch '10.3' into 10.4 2019-01-26 01:13:41 +01:00
Aleksey Midenkov
06a37d37a1 MDEV-18122 Assertion 'table->versioned() == m_prebuilt->table->versioned()' failed in ha_innobase::open
Closes 
2019-01-25 23:30:45 +02:00
Honza Horak
7334f9717d Do not import commands library as it is not used
Closes 
2019-01-25 20:00:00 +01:00
Eugene Kosov
0e1f7f5c4a MDEV-18057 Assertion `(node->state == 5) || (node->state == 6)' failed in row_upd_sec_step upon DELETE after UPDATE failed due to FK violation
The idea of the fix: reset state from previous query.

row_upd_clust_step(): reset cached index before updating a clustered index

Closes 
2019-01-25 16:35:50 +02:00
Marko Mäkelä
9bd80ada6f Merge 10.2 into 10.3 2019-01-25 16:35:13 +02:00
Teemu Ollakka
040b840de7 MDEV-15740 Backport wsrep recovery fixes from 10.4.
Clear wsrep XID in innobase_rollback_by_xid() for recovered wsrep
transaction in order to avoid resetting XID storage when rolling back
wsrep transaction during recovery.

Sort wsrep XIDs read from storage engine in ascending order and
erify that the range is continuous during crash recovery. If binlog is off,
commit all recovered transactions for continuous seqno range. This is safe
because all transactions with wsrep XID have been certified and must be
committed in the cluster. On the other hand if binlog is on, respect binlog
as a transaction coordinator in order to avoid missing transactions in binlog
that have been committed into storage engine .
2019-01-25 16:19:20 +02:00
Eugene Kosov
31d0727a10 MDEV-18235: Changes related to fsync()
Remove fil_node_t::sync_event.

I had a discussion with kernel fellows and they said it's safe to call
fsync() simultaneously at least on VFS and ext4. So initially I wanted
to disable check for recent Linux but than I realized code is buggy.

Consider a case when one thread is inside fsync() and two others are
waiting inside os_event. First thread after fsync() calls os_event_set()
which is a broadcast! So two waiting threads will awake and may call
fsync() at the same time.

One fix is to add a notify_one() functionality to os_event but I decided
to remove incorrect check completely. Note, it works for one waiting
thread but not for more than one.

IMO it's ok to avoid existing bugs but there is not too much sense in
avoiding possible(!) bugs as this code does.

fil_space_t::is_in_rotation_list(), fil_space_t::is_in_unflushed_spaces():
Replace redundant bool fields with member functions.

fil_node_t::needs_flush: Replaces fil_node_t::modification_counter and
fil_node_t::flush_counter. We need to know whether there _are_ some
unflushed writes and we do not need to know _how many_ writes.

fil_system_t::modification_counter: Remove as not needed.
Even if we needed fil_node_t::modification_counter, every file
could have its own counter that would be incremented on each write.

fil_system_t::modification_counter is a global modification counter
for all files. It was incremented on every write. But whether some
file was flushed or not is an internal fil_node_t deal/state and
this makes fil_system_t::modification_counter useless.

Closes 
2019-01-25 15:40:04 +02:00
Jan Lindström
2d60e3232d MDEV-18369: Crash at wsrep_handle_SR_rollback(THD*, THD*): Assertion `victim_thd' failed.
Call to wsrep_handle_SR_rollback was missing check that wsrep_on is true.
2019-01-25 08:05:21 +02:00