Commit graph

4222 commits

Author SHA1 Message Date
Monty
91ab42a823 Clean up and speed up interfaces for binary row logging
MDEV-21605 Clean up and speed up interfaces for binary row logging
MDEV-21617 Bug fix for previous version of this code

The intention is to have as few 'if' as possible in ha_write() and
related functions. This is done by pre-calculating once per statement the
row_logging state for all tables.

Benefits are simpler and faster code both when binary logging is disabled
and when it's enabled.

Changes:
- Added handler->row_logging to make it easy to check it table should be
  row logged. This also made it easier to disabling row logging for system,
  internal and temporary tables.
- The tables row_logging capabilities are checked once per "statements
  that updates tables" in THD::binlog_prepare_for_row_logging() which
  is called when needed from THD::decide_logging_format().
- Removed most usage of tmp_disable_binlog(), reenable_binlog() and
  temporary saving and setting of thd->variables.option_bits.
- Moved checks that can't change during a statement from
  check_table_binlog_row_based() to check_table_binlog_row_based_internal()
- Removed flag row_already_logged (used by sequence engine)
- Moved binlog_log_row() to a handler::
- Moved write_locked_table_maps() to THD::binlog_write_table_maps() as
  most other related binlog functions are in THD.
- Removed binlog_write_table_map() and binlog_log_row_internal() as
  they are now obsolete as 'has_transactions()' is pre-calculated in
  prepare_for_row_logging().
- Remove 'is_transactional' argument from binlog_write_table_map() as this
  can now be read from handler.
- Changed order of 'if's in handler::external_lock() and wsrep_mysqld.h
  to first evaluate fast and likely cases before more complex ones.
- Added error checking in ha_write_row() and related functions if
  binlog_log_row() failed.
- Don't clear check_table_binlog_row_based_result in
  clear_cached_table_binlog_row_based_flag() as it's not needed.
- THD::clear_binlog_table_maps() has been replaced with
  THD::reset_binlog_for_next_statement()
- Added 'MYSQL_OPEN_IGNORE_LOGGING_FORMAT' flag to open_and_lock_tables()
  to avoid calculating of binary log format for internal opens. This flag
  is also used to avoid reading statistics tables for internal tables.
- Added OPTION_BINLOG_LOG_OFF as a simple way to turn of binlog temporary
  for create (instead of using THD::sql_log_bin_off.
- Removed flag THD::sql_log_bin_off (not needed anymore)
- Speed up THD::decide_logging_format() by remembering if blackhole engine
  is used and avoid a loop over all tables if it's not used
  (the common case).
- THD::decide_logging_format() is not called anymore if no tables are used
  for the statement. This will speed up pure stored procedure code with
  about 5%+ according to some simple tests.
- We now get annotated events on slave if a CREATE ... SELECT statement
  is transformed on the slave from statement to row logging.
- In the original code, the master could come into a state where row
  logging is enforced for all future events if statement could be used.
  This is now partly fixed.

Other changes:
- Ensure that all tables used by a statement has query_id set.
- Had to restore the row_logging flag for not used tables in
  THD::binlog_write_table_maps (not normal scenario)
- Removed injector::transaction::use_table(server_id_type sid, table tbl)
  as it's not used.
- Cleaned up set_slave_thread_options()
- Some more DBUG_ENTER/DBUG_RETURN, code comments and minor indentation
  changes.
- Ensure we only call THD::decide_logging_format_low() once in
  mysql_insert() (inefficiency).
- Don't annotate INSERT DELAYED
- Removed zeroing pos_in_table_list in THD::open_temporary_table() as it's
  already 0
2020-03-24 21:00:03 +02:00
Monty
4ef437558a Improve update handler (long unique keys on blobs)
MDEV-21606 Improve update handler (long unique keys on blobs)
MDEV-21470 MyISAM and Aria start_bulk_insert doesn't work with long unique
MDEV-21606 Bug fix for previous version of this code
MDEV-21819 2 Assertion `inited == NONE || update_handler != this'

- Move update_handler from TABLE to handler
- Move out initialization of update handler from ha_write_row() to
  prepare_for_insert()
- Fixed that INSERT DELAYED works with update handler
- Give an error if using long unique with an autoincrement column
- Added handler function to check if table has long unique hash indexes
- Disable write cache in MyISAM and Aria when using update_handler as
  if cache is used, the row will not be inserted until end of statement
  and update_handler would not find conflicting rows.
- Removed not used handler argument from
  check_duplicate_long_entries_update()
- Syntax cleanups
  - Indentation fixes
  - Don't use single character indentifiers for arguments
2020-03-24 21:00:02 +02:00
Monty
6a9e24d046 Added support for replication for S3
MDEV-19964 S3 replication support

Added new configure options:
s3_slave_ignore_updates
"If the slave has shares same S3 storage as the master"

s3_replicate_alter_as_create_select
"When converting S3 table to local table, log all rows in binary log"

This allows on to configure slaves to have the S3 storage shared or
independent from the master.

Other thing:
Added new session variable '@@sql_if_exists' to force IF_EXIST to DDL's.
2020-03-24 21:00:02 +02:00
Marko Mäkelä
5203bc10f1 Merge 10.4 into 10.5 2020-03-21 11:37:10 +02:00
Marko Mäkelä
bd3c8f47cd Merge 10.3 into 10.4 2020-03-20 22:06:55 +02:00
Marko Mäkelä
44298e4dea Merge 10.2 into 10.3
Also, clean up the test innodb_gis.geometry a little further.
2020-03-20 18:12:17 +02:00
Marko Mäkelä
c9ec1cc751 Merge 10.1 into 10.2 2020-03-20 15:51:30 +02:00
Otto Kekäläinen
c8388de2fd Fix various spelling errors
e.g.
- dont -> don't
- occurence -> occurrence
- succesfully -> successfully
- easyly -> easily

Also remove trailing space in selected files.

These changes span:
- server core
- Connect and Innobase storage engine code
- OQgraph, Sphinx and TokuDB storage engines

Related to MDEV-21769.
2020-03-16 00:10:50 +02:00
Sergei Golubchik
0fe3d6d563 all status variables above questions MUST be ulong
see e.g. `add_to_status()`
2020-03-14 12:38:26 +01:00
Sergei Golubchik
91d1588d30 Merge branch 'github/10.5' into 10.5 2020-03-14 09:52:35 +01:00
Marko Mäkelä
d82ac8d374 MDEV-21907: Fix some -Wconversion outside InnoDB
Some .c and .cc files are compiled as part of Mariabackup.
Enabling -Wconversion for InnoDB would also enable it for
Mariabackup. The .h files are being included during
InnoDB or Mariabackup compilation.

Notably, GCC 5 (but not GCC 4 or 6 or later versions)
would report -Wconversion for x|=y when the type is
unsigned char. So, we will either write x=(uchar)(x|y)
or disable the -Wconversion warning for GCC 5.

bitmap_set_bit(), bitmap_flip_bit(), bitmap_clear_bit(), bitmap_is_set():
Always implement as inline functions.
2020-03-12 19:44:52 +02:00
Oleksandr Byelkin
fad47df995 Merge branch '10.4' into 10.5 2020-03-11 17:52:49 +01:00
Sergei Golubchik
7af733a5a2 perfschema compilation, test and misc fixes 2020-03-10 19:24:23 +01:00
Sergei Golubchik
81cffda2e6 perfschema transaction instrumentation related changes 2020-03-10 19:24:23 +01:00
Sergei Golubchik
6ded554fc2 perfschema thread instrumentation related changes 2020-03-10 19:24:23 +01:00
Sergei Golubchik
0d837e8153 perfschema table io instrumentation related changes 2020-03-10 19:24:23 +01:00
Sergei Golubchik
cea187e349 perfschema sysvar instrumentation related changes
incomplete. locking issues
2020-03-10 19:24:23 +01:00
Sergei Golubchik
d4605bc90f perfschema statement instrumentation related changes 2020-03-10 19:24:23 +01:00
Sergei Golubchik
7c58e97bf6 perfschema memory related instrumentation changes 2020-03-10 19:24:22 +01:00
Varun Gupta
b753ac066b MDEV-21580: Allow packed sort keys in sort buffer
This task deals with packing the sort key inside the sort buffer, which  would
lead to efficient usage of the memory allocated for the sort buffer.

The changes brought by this feature are
  1) Sort buffers would have sort keys of variable length
  2) The format for sort keys inside the sort buffer would look like
     |<sort_length><null_byte><key_part1><null_byte><key_part2>.......|
      sort_length is the extra bytes that are required to store the variable
      length of a sort key.
  3) When packing of sort key is done we store the ORIGINAL VALUES inside
     the sort buffer and not the STRXFRM form (mem-comparable sort keys).
  4) Special comparison function packed_keys_comparison() is introduced
     to compare 2 sort keys.

This patch also contains contributions from Sergei Petrunia.
2020-03-10 15:09:17 +05:30
Oleksandr Byelkin
980108ceeb MDEV-21833 Make slave_run_triggers_for_rbr enforce triggers to run on slave, even when there are triggers on the master
A bit changed patch of Anders Karlsson with examples added.

New parameters "ENFORCE" to slave-run-triggers-for-rbr added.
2020-03-09 22:16:43 +02:00
Monty
940fcbe73b Improved speed of optimizer trace
- Added unlikely() to optimize for not having optimizer trace enabled
- Made THD::trace_started() inline
- Added 'if (trace_enabled())' around some potentially expensive code
  (not many found)
- Added ASSERT's to ensure we don't call expensive optimizer trace calls
  if optimizer trace is not enabled
- Added length to Json_writer functions to speed up buffer writes
  when optimizer trace is enabled.
- Changed LEX_CSTRING argument handling to not send full struct to writer
  function on_add_str() functions now trusts length arguments
2020-03-09 13:49:06 +02:00
Aleksey Midenkov
193725b81e MDEV-7318 RENAME INDEX
This patch adds support of RENAME INDEX operation to the ALTER TABLE
statement. Code which determines if ALTER TABLE can be done in-place
for "simple" storage engines like MyISAM, Heap and etc. was updated to
handle ALTER TABLE ... RENAME INDEX as an in-place operation. Support
for in-place ALTER TABLE ... RENAME INDEX for InnoDB was covered by
MDEV-13301.

Syntax changes
==============

A new type of <alter_specification> is added:

<rename index clause> ::= RENAME ( INDEX | KEY ) <oldname> TO <newname>

Where <oldname> and <newname> are identifiers for old name and new
name of the index.

Semantic changes
================

The result of "ALTER TABLE t1 RENAME INDEX a TO b" is a table which
contents and structure are identical to the old version of 't1' with
the only exception index 'a' being called 'b'.

Neither <oldname> nor <newname> can be "primary". The index being
renamed should exist and its new name should not be occupied
by another index on the same table.

Related to: WL#6555, MDEV-13301
2020-03-03 13:50:33 +03:00
Aleksey Midenkov
fa8ad75439 MDEV-16290 ALTER TABLE ... RENAME COLUMN syntax
The existing syntax for renaming a column uses "ALTER TABLE ...  CHANGE"
command. This requires full column specification to rename the column.
This patch adds new syntax "ALTER TABLE ...  RENAME COLUMN", which do not
expect users to provide full column specification.  It means that the new
syntax would pick in-place or copy algorithm in the same way as that of
existing "ALTER TABLE ... CHANGE" command. The existing syntax
"ALTER TABLE ... CHANGE" will continue to work.

Syntax changes
==============

ALTER TABLE tbl_name
    [alter_specification [, alter_specification] ...]
    [partition_options]

Following is a new <alter_specification> added:

 | RENAME COLUMN <oldname> TO <newname>

Where <oldname> and <newname> are identifiers for old name and new
name of the column.

Related to: WL#10761
2020-03-03 13:50:32 +03:00
Vicențiu Ciorbaru
ad17aa110c MDEV-18650: Options deprecated in previous versions - multi_range_count
Remove deprecated system variable multi_range_count. It was ignored from 5.3.
2020-02-13 13:42:01 +02:00
Alexander Barkov
83e75b39b3 MDEV-21702 Add a data type for privileges 2020-02-11 08:10:26 +04:00
mkaruza
41bc736871 Galera GTID support
Support for galera GTID consistency thru cluster. All nodes in cluster
should have same GTID for replicated events which are originating from cluster.
Cluster originating commands need to contain sequential WSREP GTID seqno
Ignore manual setting of gtid_seq_no=X.

In master-slave scenario where master is non galera node replicated GTID is
replicated and is preserved in all nodes.

To have this - domain_id, server_id and seqnos should be same on all nodes.
Node which bootstraps the cluster, to achieve this, sends domain_id and
server_id to other nodes and this combination is used to write GTID for events
that are replicated inside cluster.

Cluster nodes that are executing non replicated events are going to have different
GTID than replicated ones, difference will be visible in domain part of gtid.

With wsrep_gtid_domain_id you can set domain_id for WSREP cluster.

Functions WSREP_LAST_WRITTEN_GTID, WSREP_LAST_SEEN_GTID and
WSREP_SYNC_WAIT_UPTO_GTID now works with "native" GTID format.

Fixed galera tests to reflect this chances.

Add variable to manually update WSREP GTID seqno in cluster

Add variable to manipulate and change WSREP GTID seqno. Next command
originating from cluster and on same thread will have set seqno and
cluster should change their internal counter to it's value.
Behavior is same as using @@gtid_seq_no for non WSREP transaction.
2020-01-29 15:06:06 +02:00
Alexander Barkov
f1e13fdc8d MDEV-21581 Helper functions and methods for CHARSET_INFO 2020-01-28 12:29:23 +04:00
Oleksandr Byelkin
f31bf6f094 Merge branch '5.5' into 10.1 2020-01-19 12:22:12 +01:00
Vicențiu Ciorbaru
409aba3d99 Improve documentation of Unique class
* size represents the size of an element in the Unique class
* full_size is used when the Unique class counts the number of
duplicates stored per element. This requires additional space per Unique
element.
2020-01-16 13:22:39 +02:00
Aleksey Midenkov
5130f5206c MDEV-20480 Obsolete internal parser for FK in InnoDB
Currently InnoDB uses internal parser for adding foreign keys. Remove
internal parser and use data parsed by SQL parser (sql_yacc) for
adding foreign keys.

- create_table_info_t::create_foreign_keys() replacement for
  dict_create_foreign_constraints_low();
- Pass constraint name via Foreign_key object.

Temporary until MDEV-20865:

- Pass alter_info as part of create_info.
2019-11-20 13:18:31 +03:00
Marko Mäkelä
ae90f8431b Merge 10.4 into 10.5 2019-11-14 14:49:20 +02:00
Sujatha
caa79081c3 MDEV-20707: Missing memory barrier in parallel replication error handler in wait_for_prior_commit()
revision-id: 673e253724979fd9fe43a4a22bd7e1b2c3a5269e
Author: Kristian Nielsen

Fix missing memory barrier in wait_for_commit.

The function wait_for_commit::wait_for_prior_commit() has a fast path where it
checks without locks if wakeup_subsequent_commits() has already been called.
This check was missing a memory barrier. The waitee thread does two writes to
variables `waitee' and `wakeup_error', and if the waiting thread sees the
first write it _must_ also see the second or incorrect behavior will occur.
This requires memory barriers between both the writes (release semantics) and
the reads (acquire semantics) of those two variables.

Other accesses to these variables are done under lock or where only one thread
will be accessing them, and can be done without barriers (relaxed semantics).
2019-11-14 12:03:39 +05:30
Sergei Petrunia
68ed3a81f2 MDEV-20854: ANALYZE for statements: not clear where the time is spent
Count the "gap" time between table accesses and display it as
r_other_time_ms in the "table" element.

* The advantage of this approach is that it doesn't add any new
  my_timer_cycles() calls.
* The disadvantage is that the definition of what is done during
  "other time" is not that clear: it includes checking the WHERE
  (for this table), constructing index lookup tuple (for the next table)
  writing to GROUP BY temporary table (as we dont account for that time
  separately [yet], etc)
2019-11-12 14:40:00 +03:00
Oleksandr Byelkin
3ad37ed0eb Merge 10.4 into 10.5 2019-11-07 08:52:30 +01:00
Marko Mäkelä
ec40980ddd Merge 10.3 into 10.4 2019-11-01 15:23:18 +02:00
Oleksandr Byelkin
55b2281a5d Merge branch '10.2' into 10.3 2019-10-31 10:58:06 +01:00
Sergei Golubchik
fd6dfb3b54 Merge branch 'github/10.1' into 10.2 2019-10-30 23:38:05 +01:00
Sergei Golubchik
5392b4a32c MDEV-20354 All but last insert ignored in InnoDB tables when table locked
mysql_insert() first opens all affected tables (which implicitly
starts a transaction in InnoDB), then stat tables.
A failure to open a stat table caused open_tables() to abort
the current stmt transaction (trans_rollback_stmt()). So, from the
server point of view the following ha_write_row()-s happened outside
of a transactions, and the server didn't bother to commit them.

The server has a mechanism to prevent a transaction being
unexpectedly committed or rolled back in the middle of a statement -
if an operation takes place _in a sub-statement_ it cannot change
the transaction state. Operations on stat tables are exactly that -
they are not allowed to change a transaction state. Put them in
a sub-statement to make sure they don't.
2019-10-30 23:14:44 +01:00
Sergei Golubchik
42ada91542 cleanup: RAII helper for swapping of thd->security_ctx 2019-10-28 08:17:56 +01:00
Monty
67687d06bf Simplify TABLE::decide_logging_format()
- Use local variables table and share to simplify code
- Use sql_command_flags to detect what kind of command was used
- Added CF_DELETES_DATA to simplify detecton of delete commands
- Removed duplicate error in create_table_from_items().
2019-10-20 11:52:29 +03:00
Rucha Deodhar
837ad9ab97 MDEV-10014 Add RETURNING to INSERT
Closes 
2019-10-14 10:29:31 +02:00
Sergei Golubchik
721a9df751 cleanup: formatting
comments, whitespaces
2019-10-14 10:29:31 +02:00
Oleksandr Byelkin
a6de640804 MDEV-18553: MDEV-16327 pre-requisits part 3: move kill check in one place
Kill check moved from send_data() methids in its wrapper:
send_data_with_check().
2019-10-13 09:40:41 +02:00
Oleksandr Byelkin
1ae02f0e0d MDEV-18553: MDEV-16327 pre-requisits part 2: uniform of LIMIT/OFFSET handling
Now both offset and limit are stored and do not chenged during execution
(offset is decreased during processing in versions before 10.5).

(Big part of this changes made by Monty)
2019-10-13 09:40:41 +02:00
Oleksandr Byelkin
eb0804ef5e MDEV-18553: MDEV-16327 pre-requisits part 1: isolation of LIMIT/OFFSET handling 2019-10-13 09:40:41 +02:00
Alexander Barkov
530f3f7cfc MDEV-20806 Federated does not work with INET6, returns NULL with warning ER_TRUNCATED_WRONG_VALUE 2019-10-12 07:25:53 +04:00
Marko Mäkelä
d04f2de80a Merge 10.4 into 10.5 2019-10-11 08:41:36 +03:00
Marko Mäkelä
c11e5cdd12 Merge 10.3 into 10.4 2019-10-10 11:19:25 +03:00
Marko Mäkelä
892378fb9d Merge 10.2 into 10.3 2019-10-09 13:25:11 +03:00