Commit graph

2005 commits

Author SHA1 Message Date
Nikita Malyavin
6294516a56 MDEV-16975 Application-time periods: ALTER TABLE
* implicit period constraint is hidden and cannot be dropped independently
* create...like and create...select support
2019-02-21 14:57:09 +01:00
Nikita Malyavin
073c93b194 MDEV-17082 Application-time periods: CREATE
* add syntax `CREATE TABLE ... PERIOD FOR <apptime>`
* add table period entity
2019-02-21 14:48:04 +01:00
Marko Mäkelä
2c74799d64 MDEV-18627 Wrong result after instant size change of integer
If we instantly change the size of a fixed-length field
and treat it as kind-of variable-length, then we will need
conversions between old column values and new ones.

I tried adding such a conversion to row_build(), but then I
noticed that more conversions would be needed, because
old values still appeared in a freshly rebuilt secondary index,
causing a mismatch when trying to search with the correct
longer value that was converted in my provisional fix to row_build().

So, we will revert the essential part of
MDEV-15563: Instant ROW_FORMAT=REDUNDANT column extension
(commit 22feb179ae), but not
remove any tests.
2019-02-18 18:30:01 +02:00
Igor Babaev
98d55b1366 Merge branch '10.4' into bb-10.4-mdev16188 2019-02-14 22:07:33 -08:00
Marko Mäkelä
ea0be9e2d6 MDEV-15564 Avoid table rebuild in ALTER TABLE on collation or charset changes
Allow ALGORITHM=INSTANT (or avoid touching any data)
when changing the collation, or in some cases, the character set,
of a non-indexed CHAR or VARCHAR column. There is no penalty
for subsequent DDL or DML operations, and compatibility with
older MariaDB versions will be unaffected.

Character sets may be changed when the old encoding is compatible
with the new one. For example, changing from ASCII to anything
ASCII-based, or from 3-byte to 4-byte UTF-8 can sometimes be
performed instantly.

This is joint work with Eugene Kosov.
The test cases as well as ALTER_CONVERT_TO, charsets_are_compatible(),
Type_handler::Charsets_are_compatible() are his work.
The Field_str::is_equal(), Field_varstring::is_equal() and
the InnoDB changes were mostly rewritten by me due to conflicts
with MDEV-15563.

Limitations:

Changes of indexed columns will still require
ALGORITHM=COPY. We should allow ALGORITHM=NOCOPY and allow
the indexes to be rebuilt inside the storage engine,
without copying the entire table.

Instant column size changes (in bytes) are not supported by
all storage engines.

Instant CHAR column changes are only allowed for InnoDB
ROW_FORMAT=REDUNDANT. We could allow this for InnoDB
when the CHAR internally uses a variable-length encoding,
say, when converting from 3-byte UTF-8 to 4-byte UTF-8.

Instant VARCHAR column changes are allowed for InnoDB
ROW_FORMAT=REDUNDANT, and for others only if the size
in bytes does not change from 128..255 bytes to more
than 256 bytes.

Inside InnoDB, this slightly changes the way how MDEV-15563
works and fixes the result of the innodb.instant_alter_extend test.
We change the way how ALTER_COLUMN_EQUAL_PACK_LENGTH_EXT
is handled. All column extension, type changes and renaming
now go through a common route, except when ctx->is_instant()
is in effect, for example, instant ADD or DROP COLUMN has
been initiated. Only in that case we will go through
innobase_instant_try() and rewrite all column metadata.

get_type(field, prtype, mtype, len): Convert a SQL data type into
InnoDB column metadata.

innobase_rename_column_try(): Remove the update of SYS_COLUMNS.

innobase_rename_or_enlarge_column_try(): New function,
replacing part of innobase_rename_column_try() and all of
innobase_enlarge_column_try(). Also changes column types.

innobase_rename_or_enlarge_columns_cache(): Also change
the column type.
2019-02-14 20:17:14 +02:00
Igor Babaev
7d9f45e072 Merge branch '10.4' into bb-10.4-mdev17096 2019-02-13 14:59:34 -08:00
Marko Mäkelä
22feb179ae MDEV-15563: Instant ROW_FORMAT=REDUNDANT column extension
This was developed by Aleksey Midenkov based on my design.

In the original InnoDB storage format (that was retroactively named
ROW_FORMAT=REDUNDANT in MySQL 5.0.3), the length of each index field
is stored explicitly.

Because of this, we can and now will allow instant conversion from
VARCHAR to CHAR or VARBINARY to BINARY of equal or greater size,
as well as instant conversion of TINYINT to SMALLINT to MEDIUMINT
to INT to BIGINT (while not changing between signed and unsigned).

Theoretically, we could allow changing from an unsigned integer to
a bigger unsigned integer, as well as changing CHAR to VARCHAR, but
that would require additional metadata and conversions whenever
reading old records.

Field_str::is_equal(), Field_varstring::is_equal(), Field_num::is_equal():
Return the new result IS_EQUAL_PACK_LENGTH_EXT if the table advertises
HA_EXTENDED_TYPES_CONVERSION capability and we are considering the
above-mentioned conversions.

ALTER_COLUMN_EQUAL_PACK_LENGTH_EXT: A new ALTER TABLE flag, similar
to ALTER_COLUMN_EQUAL_PACK_LENGTH but requiring conversions when
reading the data. The Field::is_equal() result IS_EQUAL_PACK_LENGTH_EXT
will map to this flag.

dtype_get_fixed_size_low(): For BINARY, CHAR and integer columns
in ROW_FORMAT=REDUNDANT, return 0 (variable length) from now on.

dtype_get_sql_null_size(): Keep returning the current size for
BINARY, CHAR and integer columns, so that in ROW_FORMAT=REDUNDANT
it will remain possible to update in place between NULL and NOT NULL
values.

btr_index_rec_validate(): Relax a CHECK TABLE length check for
ROW_FORMAT=REDUNDANT tables.

btr_cur_instant_init_low(): No longer trust fixed_len
for ROW_FORMAT=REDUNDANT tables.

We cannot rely on fixed_len anymore because the record can have shorter
length from before instant extension. Note that importing such tablespace
into earlier MariaDB versions produces ER_TABLE_SCHEMA_MISMATCH when
using a .cfg file.
2019-02-13 17:39:05 +02:00
Marko Mäkelä
0ae3ea7919 MDEV-15563: Instant VARCHAR extension for ROW_FORMAT=REDUNDANT
In the original InnoDB storage format (which was retroactively named
ROW_FORMAT=REDUNDANT in MySQL 5.0.3), the length of each index field
is stored explicitly. Thus, we can and from now on will allow arbitrary
extension of VARBINARY and VARCHAR columns when the table is in
ROW_FORMAT=REDUNDANT.

ha_innobase::open(): Advertise a new HA_EXTENDED_TYPES_CONVERSION
capability for ROW_FORMAT=REDUNDANT tables.

Field_varstring::is_equal(): If the HA_EXTENDED_TYPES_CONVERSION
capability is advertised for the table, return IS_EQUAL_PACK_LENGTH
for any length extension.
2019-02-13 16:42:03 +02:00
Igor Babaev
953ca199fb Merge branch '10.4' into bb-10.4-mdev17096 2019-02-12 23:19:43 -08:00
Varun Gupta
be8709eb7b MDEV-6111 Optimizer Trace
This task involves the implementation for the optimizer trace.

This feature produces a trace for any SELECT/UPDATE/DELETE/,
which contains information about decisions taken by the optimizer during
the optimization phase (choice of table access method, various costs,
transformations, etc). This feature would help to tell why some decisions were
taken by the optimizer and why some were rejected.

Trace is session-local, controlled by the @@optimizer_trace variable.
To enable optimizer trace we need to write:
   set @@optimizer_trace variable= 'enabled=on';

To display the trace one can run:
   SELECT trace FROM INFORMATION_SCHEMA.OPTIMIZER_TRACE;

This task also involves:
    MDEV-18489: Limit the memory used by the optimizer trace
    introduces a switch optimizer_trace_max_mem_size which limits
    the memory used by the optimizer trace. This was implemented by
    Sergei Petrunia.
2019-02-13 11:52:36 +05:30
Igor Babaev
651347b6c1 MDEV-16188 Post merge fixes fot TokuDB 2019-02-08 01:07:27 -08:00
Igor Babaev
3f9040085a Merge branch '10.4' into bb-10.4-mdev17096 2019-02-06 18:01:29 -08:00
Igor Babaev
16327fc2e7 MDEV-17096 Pushdown of simple derived tables to storage engines
MDEV-17631 select_handler for a full query pushdown

Interfaces + Proof of Concept for federatedx with test cases.

The interfaces have been developed for integration of ColumnStore engine.
2019-02-06 17:02:44 -08:00
Galina Shalygina
447e0f023f MDEV-18144: ANALYZE for statement support for PK filters
ANALYZE and ANALYZE FORMAT=JSON structures are changed in the way that they
show additional information when rowid filter is used:

- r_selectivity_pct - the observed filter selectivity
- r_buffer_size - the size of the rowid filter container buffer
- r_filling_time_ms - how long it took to fill rowid filter container

New class Rowid_filter_tracker was added. This class is needed to collect data
about how rowid filter is executed.
2019-02-06 23:40:07 +03:00
Marko Mäkelä
e80bcd7f64 Merge 10.3 into 10.4 2019-02-05 12:48:02 +02:00
Marko Mäkelä
ab2458c61f Merge 10.2 into 10.3 2019-02-04 15:12:14 +02:00
Igor Babaev
37deed3f37 Merge branch '10.4' into bb-10.4-mdev16188 2019-02-03 18:41:18 -08:00
Igor Babaev
658128af43 MDEV-16188 Use in-memory PK filters built from range index scans
This patch contains a full implementation of the optimization
that allows to use in-memory rowid / primary filters built for range  
conditions over indexes. In many cases usage of such filters reduce  
the number of disk seeks spent for fetching table rows.

In this implementation the choice of what possible filter to be applied  
(if any) is made purely on cost-based considerations.

This implementation re-achitectured the partial implementation of
the feature pushed by Galina Shalygina in the commit
8d5a11122c.

Besides this patch contains a better implementation of the generic  
handler function handler::multi_range_read_info_const() that
takes into account gaps between ranges when calculating the cost of
range index scans. It also contains some corrections of the
implementation of the handler function records_in_range() for MyISAM.

This patch supports the feature for InnoDB and MyISAM.
2019-02-03 14:56:12 -08:00
Thirunarayanan Balathandayuthapani
b8aef87221 MDEV-16849 Extending indexed VARCHAR column should be instantaneous
Analysis:
========
Increasing the length of the indexed varchar column is not an instant operation for
innodb.

Fix:
===
- Introduce the new handler flag 'Alter_inplace_info::ALTER_COLUMN_INDEX_LENGTH' to
indicate the index length differs due to change of column length changes.

- InnoDB makes the ALTER_COLUMN_INDEX_LENGTH flag as instant operation.

This is a port of Mysql fix.

    commit 913071c0b16cc03e703308250d795bc381627e37
    Author: Nisha Gopalakrishnan <nisha.gopalakrishnan@oracle.com>
    Date:   Wed May 30 14:54:46 2018 +0530

        BUG#26848813: INDEXED COLUMN CAN'T BE CHANGED FROM VARCHAR(15)
                      TO VARCHAR(40) INSTANTANEOUSLY
2019-01-30 15:33:32 +05:30
Brave Galera Crew
36a2a185fe Galera4 2019-01-23 15:30:00 +04:00
Anel Husakovic
802e8d6b89 Backport INFORMATION_SCHEMA.CHECK_CONSTRAINTS
Implement according to standard SQL specification 2008.
The check_constraints table is used for fetching metadata about
the constraints defined for tables in all databases.

There were some result files which failed after running mtr.
These files are updated with newly create record with mtr --record.
2019-01-16 13:34:54 +02:00
Marko Mäkelä
734510a44d Merge 10.3 into 10.4 2019-01-06 17:43:02 +02:00
Sergei Golubchik
6bb11efa4a Merge branch '10.2' into 10.3 2019-01-03 13:09:41 +01:00
Sergei Golubchik
67240858b2 Merge branch '10.1' into 10.2 2018-12-30 18:30:29 +01:00
Sergei Golubchik
aeefd26ecb Merge branch '10.0' into 10.1 2018-12-29 23:44:45 +01:00
Nikita Malyavin
6a73569f12 MDEV-16429: Assertion `!table || (!table->read_set || bitmap_is_set(table->read_set, field_index))' fails upon attempt to update virtual column on partitioned versioned table
When using buffered sort in `UPDATE`, keyread is used. In this case,
`TABLE::update_virtual_field` should be aborted, but it actually isn't,
because it is called not with a top-level handler, but with the one that
is actually going to access the disk. Here the problemm is issued with
partitioning, so the solution is to recursively mark for keyread all the
underlying partition handlers.

* ha_partition: update keyread state for child partitions

Closes 
2018-12-20 08:06:55 +01:00
Varun Gupta
7e606a2d5c MDEV-17589: Stack-buffer-overflow with indexed varchar (utf8) field
Create a new constant MAX_DATA_LENGTH_FOR_KEY.
Replace the value of MAX_KEY_LENGTH to also include the LENGTH and NULL BYTES
of a field.
2018-12-19 10:38:46 +05:30
Monty
965311ee8b Added new MDL_BACKUP locks for all backup stages
Part of MDEV-5336 Implement LOCK FOR BACKUP

- Added new locks to MDL_BACKUP for all stages of backup locks and
  a new MDL lock needed for backup stages.
- Renamed MDL_BACKUP_STMT to MDL_BACKUP_DDL
- flush_tables() takes a new parameter that decides what should be flushed.
- InnoDB, Aria (transactional tables with checksums), Blackhole, Federated
  and Federatedx tables are marked to be safe for online backup. We are
  using MDL_BACKUP_TRANS_DML instead of MDL_BACKUP_DML locks for these
  which allows any DML's to proceed for these tables during the whole
  backup process until BACKUP STAGE COMMIT which will block the final
  commit.
2018-12-09 22:12:27 +02:00
Monty
1077f320e4 Added backup handler calls
Part of MDEV-5336 Implement LOCK FOR BACKUP

Added support for backup calls in Aria to protect
removal of redo logs.
2018-12-09 22:12:25 +02:00
Sergei Golubchik
f0f0d07250 MDEV-14500 filesort to support engines with slow rnd_pos
If the engine wants to avoid rnd_pos() - force a temporary table
before a filesort. But don't do it if addon_fields are used.
2018-11-20 15:06:03 +01:00
Sergei Golubchik
1823ce7304 cleanup: rename a flag, keep old name for compatibility 2018-11-20 15:05:52 +01:00
Sergei Golubchik
649465db41 cleanup: remove HA_WANTS_PRIMARY_KEY as redundant
We don't have many bits left, no need to add another InnoDB-specific flag.
Instead, we say that HA_REQUIRE_PRIMARY_KEY does not apply to SEQUENCE.
Meaning, if the engine declares HA_CAN_TABLES_WITHOUT_ROLLBACK (required
for SEQUENCE) it *must* support tables without a primary key.
2018-11-20 15:05:46 +01:00
Igor Babaev
5f46670bd0 Merge branch '10.4' into 10.4-mdev16188 2018-11-10 14:52:57 -08:00
Marko Mäkelä
074c684099 Merge 10.3 into 10.4 2018-11-06 16:24:16 +02:00
Monty
d30124e844 MDEV-17503 CREATE SEQUENCE failed with innodb_force_primary_key =1
Fixed by adding table flag HA_WANTS_PRIMARY_KEY, which is like
HA_REQUIRE_PRIMARY_KEY but tells SQL upper layer that the storage engine
internally can handle tables without primary keys (for example for
sequences or trough user variables)
2018-10-29 16:12:52 +02:00
Monty
6a6cc8a653 Remove not used table_flag HA_NO_VARCHAR 2018-10-29 15:48:49 +02:00
Alexander Barkov
3e47b41a16 MDEV-17542 10.3: gcc-8.0 produces lots of -Wclass-memaccess warnings in Table_scope_and_contents_source_st 2018-10-25 14:07:31 +04:00
Marko Mäkelä
444c380ceb Merge 10.3 into 10.4 2018-10-05 08:09:49 +03:00
Galina Shalygina
8d5a11122c MDEV-16188: Use in-memory PK filters built from range index scans
First phase: make optimizer choose to use filter and show it in EXPLAIN.
2018-09-28 23:50:22 +03:00
Nikita Malyavin
c16a54c02e MDEV-16429: Assertion `!table || (!table->read_set || bitmap_is_set(table->read_set, field_index))' fails upon attempt to update virtual column on partitioned versioned table
When using buffered sort in `UPDATE`, keyread is used. In this case,
`TABLE::update_virtual_field` should be aborted, but it actually isn't,
because it is called not with a top-level handler, but with the one that
is actually going to access the disk. Here the problemm is issued with
partitioning, so the solution is to recursively mark for keyread all the
underlying partition handlers.

* ha_partition: update keyread state for child partitions

Closes 
2018-09-21 15:05:54 +02:00
Marko Mäkelä
1bf3e8ab43 Merge 10.3 into 10.4 2018-09-11 21:31:03 +03:00
Anel Husakovic
d526679efd MDEV-14474 information_schema.check_constraints
Implement according to standard SQL specification 2008.
The check_constraints table is used for fetching metadata about
the constraints defined for tables in all databases.
2018-08-25 17:03:20 +03:00
Marko Mäkelä
734db318ac Merge 10.3 into 10.4 2018-08-16 10:08:30 +03:00
Michael Widenius
d6d63f4844 MDEV-16421 Make system tables crash safe
Make all system tables in mysql directory of type
engine=Aria

Privilege tables are using transactional=1
Statistical tables are using transactional=0, to allow them
to be quickly updated with low overhead.
Help tables are also using transactional=0 as these are only
updated at init time.

Other changes:
- Aria store engine is now a required engine
- Update comment for Aria tables to reflect their new usage
- Fixed that _ma_reset_trn_for_table() removes unlocked table
  from transaction table list. This was needed to allow one
  to lock and unlock system tables separately from other
  tables, for example when reading a procedure from mysql.proc
- Don't give a warning when using transactional=1 for engines
  that is using transactions. This is both logical and also
  to avoid warnings/errors when doing an alter of a privilege
  table to InnoDB.
- Don't abort on warnings from ALTER TABLE for changes that
  would be accepted by CREATE TABLE.
- New created Aria transactional tables are marked as not movable
  (as they include create_rename_lsn).
- bootstrap.test was changed to kill orignal server, as one
  can't anymore have two servers started at same time on same
  data directory and data files.
- Disable maria.small_blocksize as one can't anymore change
  aria block size after system tables are created.
- Speed up creation of help tables by using lock tables.
- wsrep_sst_resync now also copies Aria redo logs.
2018-08-14 12:18:38 +03:00
Marko Mäkelä
05459706f2 Merge 10.2 into 10.3 2018-08-03 15:57:23 +03:00
Marko Mäkelä
814ae57daf Merge 10.1 into 10.2 2018-08-03 13:02:56 +03:00
Marko Mäkelä
769f6d2db7 Fix -Wclass-memaccess in WSREP,InnoDB,XtraDB 2018-08-03 12:21:13 +03:00
Jacob Mathew
d6594847cf MDEV-16246: insert timestamp into spider table from mysqldump gets wrong time zone.
The problem occurred because the Spider node was incorrectly handling
timestamp values sent to and received from the data nodes.

The problem has been corrected as follows:
- Added logic to set and maintain the UTC time zone on the data nodes.
  To prevent timestamp ambiguity, it is necessary for the data nodes to use
  a time zone such as UTC which does not have daylight savings time.
- Removed the spider_sync_time_zone configuration variable, which did not
  solve the problem and which interfered with the solution.
- Added logic to convert to the UTC time zone all timestamp values sent to
  and received from the data nodes.  This is done for both unique and
  non-unique timestamp columns.  It is done for WHERE clauses, applying to
  SELECT, UPDATE and DELETE statements, and for UPDATE columns.
- Disabled Spider's use of direct update when any of the columns to update is
  a timestamp column.  This is necessary to prevent false duplicate key value
  errors.
- Added a new test spider.timestamp to thoroughly test Spider's handling of
  timestamp values.

Author:
  Jacob Mathew.

Reviewer:
  Kentoku Shiba.

Merged:
  Commit 97cc9d3 on branch bb-10.3-MDEV-16246
2018-07-24 15:57:13 -07:00
Jacob Mathew
813b739850 MDEV-16246: insert timestamp into spider table from mysqldump gets wrong time zone.
The problem occurred because the Spider node was incorrectly handling
timestamp values sent to and received from the data nodes.

The problem has been corrected as follows:
- Added logic to set and maintain the UTC time zone on the data nodes.
  To prevent timestamp ambiguity, it is necessary for the data nodes to use
  a time zone such as UTC which does not have daylight savings time.
- Removed the spider_sync_time_zone configuration variable, which did not
  solve the problem and which interfered with the solution.
- Added logic to convert to the UTC time zone all timestamp values sent to
  and received from the data nodes.  This is done for both unique and
  non-unique timestamp columns.  It is done for WHERE clauses, applying to
  SELECT, UPDATE and DELETE statements, and for UPDATE columns.
- Disabled Spider's use of direct update when any of the columns to update is
  a timestamp column.  This is necessary to prevent false duplicate key value
  errors.
- Added a new test spider.timestamp to thoroughly test Spider's handling of
  timestamp values.

Author:
  Jacob Mathew.

Reviewer:
  Kentoku Shiba.

Cherry-Picked:
  Commit 97cc9d3 on branch bb-10.3-MDEV-16246
2018-07-09 16:09:20 -07:00
Sergei Golubchik
36e59752e7 Merge branch '10.2' into 10.3 2018-06-30 16:39:20 +02:00