Commit graph

188756 commits

Author SHA1 Message Date
Otto Kekäläinen
c8388de2fd Fix various spelling errors
e.g.
- dont -> don't
- occurence -> occurrence
- succesfully -> successfully
- easyly -> easily

Also remove trailing space in selected files.

These changes span:
- server core
- Connect and Innobase storage engine code
- OQgraph, Sphinx and TokuDB storage engines

Related to MDEV-21769.
2020-03-16 00:10:50 +02:00
Vladislav Vaintroub
3c57693ff1 MDEV-21534 - Improve innodb redo log group commit performance
Instrument new synchronization primitive with thd_wait_begin/end
to inform threadpool about waits.

This considerably improve performance on write benchmarks
(e.g sysbench update_index) with generic threadpool, of course the cost is
possibility of many newly created threads.
2020-03-15 21:40:11 +01:00
Varun Gupta
637c17588a MDEV-21922: Allow packing addon fields even if they don't honour max_length_for_sort_data
Addon fields will be packed if the length of addon fields is greater
than max_length_for_sort_data.
2020-03-15 23:48:51 +05:30
Andrei Elkin
345e21d2eb MDEV-742 test correction.
Fixed XA-ROLLBACK not to write into binlog when XA-prepare run
in sql_log_bin-OFF environment. (Two tests affected).
2020-03-15 14:14:35 +02:00
Sergey Vojtovich
8e5ae4e4df Test case fix: sort XA RECOVER result
Part of
MDEV-742 - XA PREPAREd transaction survive disconnect/server restart
2020-03-15 12:52:53 +04:00
Andrei Elkin
c8ae357341 MDEV-742 XA PREPAREd transaction survive disconnect/server restart
Lifted long standing limitation to the XA of rolling it back at the
transaction's
connection close even if the XA is prepared.

Prepared XA-transaction is made to sustain connection close or server
restart.
The patch consists of

    - binary logging extension to write prepared XA part of
      transaction signified with
      its XID in a new XA_prepare_log_event. The concusion part -
      with Commit or Rollback decision - is logged separately as
      Query_log_event.
      That is in the binlog the XA consists of two separate group of
      events.

      That makes the whole XA possibly interweaving in binlog with
      other XA:s or regular transaction but with no harm to
      replication and data consistency.

      Gtid_log_event receives two more flags to identify which of the
      two XA phases of the transaction it represents. With either flag
      set also XID info is added to the event.

      When binlog is ON on the server XID::formatID is
      constrained to 4 bytes.

    - engines are made aware of the server policy to keep up user
      prepared XA:s so they (Innodb, rocksdb) don't roll them back
      anymore at their disconnect methods.

    - slave applier is refined to cope with two phase logged XA:s
      including parallel modes of execution.

This patch does not address crash-safe logging of the new events which
is being addressed by MDEV-21469.

CORNER CASES: read-only, pure myisam, binlog-*, @@skip_log_bin, etc

Are addressed along the following policies.
1. The read-only at reconnect marks XID to fail for future
   completion with ER_XA_RBROLLBACK.

2. binlog-* filtered XA when it changes engine data is regarded as
   loggable even when nothing got cached for binlog.  An empty
   XA-prepare group is recorded. Consequent Commit-or-Rollback
   succeeds in the Engine(s) as well as recorded into binlog.

3. The same applies to the non-transactional engine XA.

4. @@skip_log_bin=OFF does not record anything at XA-prepare
   (obviously), but the completion event is recorded into binlog to
   admit inconsistency with slave.

The following actions are taken by the patch.

At XA-prepare:
   when empty binlog cache - don't do anything to binlog if RO,
   otherwise write empty XA_prepare (assert(binlog-filter case)).

At Disconnect:
   when Prepared && RO (=> no binlogging was done)
     set Xid_cache_element::error := ER_XA_RBROLLBACK
     *keep* XID in the cache, and rollback the transaction.

At XA-"complete":
   Discover the error, if any don't binlog the "complete",
   return the error to the user.

Kudos
-----
Alexey Botchkov took to drive this work initially.
Sergei Golubchik, Sergei Petrunja, Marko Mäkelä provided a number of
good recommendations.
Sergei Voitovich made a magnificent review and improvements to the code.
They all deserve a bunch of thanks for making this work done!
2020-03-14 22:45:48 +02:00
Monty
5754ea2eca Fixed compiler failures with gcc 7.4.1 and new my_malloc code 2020-03-14 15:24:13 +02:00
Sergei Golubchik
41cba6c90b MDEV-21920 binlog_encryption.rpl_gtid_basic test failure with --ps
revert e74c1c9ece to restore 2bbcf9a19c
2020-03-14 09:56:00 +01:00
Sergei Golubchik
91d1588d30 Merge branch 'github/10.5' into 10.5 2020-03-14 09:52:35 +01:00
Sergei Golubchik
c4f3e37b0c fix a race condition in the perfschema.transaction_nested_events 2020-03-14 09:51:52 +01:00
Sergei Golubchik
92c05a391a fix a race condition in the main.grant_kill test
also remove redundant --connection commands
2020-03-14 09:51:52 +01:00
Sergei Golubchik
df25d67d5f perfschema test formatting. Use --echo # 2020-03-14 09:51:36 +01:00
Sergei Golubchik
57de4def85 MDEV-21222 mariabackup.incremental_backup failed with memory allocation failure
mariabackup tries to allocate a buffer of page_size*page_size/4 size.
for 64k page it means 1Gb, which doesn't work very well on 32-bit builders.

Skip the 64k page test on 32bit.
2020-03-14 09:51:36 +01:00
Sergei Golubchik
422ba20591 mtr sets MYSQLTEST_REAL_VARDIR when MYSQLTEST_VARDIR is a symlink
this fixes main.mysqld--defaults-file failure with --mem
2020-03-14 09:51:36 +01:00
Eugene Kosov
774fe8969a cleanup redo log 2020-03-14 00:52:21 +03:00
Sergey Vojtovich
78cc9c9ebf Pre-MDEV-742 InnoDB fixes
1. Refactored innobase_close_connection(). Transaction must've already
been rolled back by this time. We should expect only transactions in the
PREPARED state when MDEV-742 is done.

2. Added missing put_pins() to trx_disconnect_prepared(). Missing
put_pins() wasn't a problem because trx_disconnect_prepared() is a dead
code. But it will get revived in the main MDEV-742 patch.

3. Fixed missing reset of trx->mysql_log_file_name when RW transaction
didn't emit any log records (zero-modification RW). The problem was
detected by ASAN when disconnected XA transaction was trying to make
use of inherited mysql_log_file_name pointing into binlog data of
detached THD.

This missing reset also had user-visible side effect, when
trx_sys_print_mysql_binlog_offset() would report binlog position
not of the most recently committed transaction.

One of possible scenarios that is expected to misbehave is as following:

  thr1> CREATE TABLE t1(a INT) ENGINE=InnoDB;
  thr1> INSERT INTO t1 VALUES(1);
  thr1> BEGIN;
  thr1> UPDATE t1 SET a=1
  thr1> COMMIT; -- zero-modification, misses to reset mysql_log_file_name

  thr2> BEGIN;
  thr2> INSERT INTO t1 VALUES(2);
  thr2> COMMIT;

  thr1> BEGIN;
  thr1> do-some-real-changes;
  thr1> ROLLBACK; -- will store binlog pos from previous COMMIT in thr1?

In this case it means if binlog is replayed from position reported by
trx_sys_print_mysql_binlog_offset(), t1 will end up with two records
containing '2'.

Part of
MDEV-742 - XA PREPAREd transaction survive disconnect/server restart
2020-03-13 15:44:42 +04:00
Sergey Vojtovich
662d8a8638 Extended debug_sync_control life time
It now lives from THD constructor to THD destructor. Reset before THD is
released to a cache. Change user doesn't reset debug_sync_control anymore.

Needed to be able to make use of DEBUG_SYNC() at later stages like
ha_close_connection().
2020-03-13 15:44:42 +04:00
Thirunarayanan Balathandayuthapani
c58686447f MDEV-21903 FTS optimize thread aborts during shutdown
- This issue was caused by 5e62b6a5e0.
fts_optimize_callback() should free fts_optimize_wq and make it as NULL
when it receives FTS_MSG_STOP message. So that subsequent
fts_optimize_callback() doesn't fail with segmentation fault.
2020-03-13 13:52:07 +05:30
Marko Mäkelä
e74c1c9ece Update libmariadb to fix GCC -Wstringop-truncation 2020-03-13 10:12:32 +02:00
Marko Mäkelä
fbe662a503 MDEV-15058: Remove buf_pool_get_dirty_pages_count()
Starting with commit 1a6f708ec5
the function buf_pool_get_dirty_pages_count() is only used
in a debug check. It was dead code for non-debug builds.

buf_flush_dirty_pages(): Perform the debug check inline,
and replace the assertion
	ut_ad(first || buf_pool_get_dirty_pages_count(id) == 0);
with another one that is executed while holding the mutexes:
	ut_ad(id != bpage->id.space());
2020-03-13 10:09:15 +02:00
Alexander Barkov
2bbcf9a19c MDEV-21920 binlog_encryption.rpl_gtid_basic test failure with --ps
Updating libmariadb to use 3be5897c3346639fa6d7195480d93108798c4917,
which fixed the problem.
2020-03-13 09:09:21 +04:00
Marko Mäkelä
f48718be4e MDEV-20632: Fix -Wmaybe-uninitialized
Create_tmp_table::add_fields(): Initialize uneven_delta= 0
to suppress the warning from GCC 9.2.1 and 10.0.1,
and consistently indent the code with spaces.
2020-03-13 06:59:34 +02:00
Marko Mäkelä
c57b207958 MDEV-21907: Fix or disable -Wconversion on GCC 5.3.0 i386
Fix or disable those -Wconversion that were missed by GCC 5.4.0
targeting AMD64.
2020-03-13 06:55:00 +02:00
Marko Mäkelä
f224525204 MDEV-21907: InnoDB: Enable -Wconversion on clang and GCC
The -Wconversion in GCC seems to be stricter than in clang.
GCC at least since version 4.4.7 issues truncation warnings for
assignments to bitfields, while clang 10 appears to only issue
warnings when the sizes in bytes rounded to the nearest integer
powers of 2 are different.

Before GCC 10.0.0, -Wconversion required more casts and would not
allow some operations, such as x<<=1 or x+=1 on a data type that
is narrower than int.

GCC 5 (but not GCC 4, GCC 6, or any later version) is complaining
about x|=y even when x and y are compatible types that are narrower
than int.  Hence, we must rewrite some x|=y as
x=static_cast<byte>(x|y) or similar, or we must disable -Wconversion.

In GCC 6 and later, the warning for assigning wider to bitfields
that are narrower than 8, 16, or 32 bits can be suppressed by
applying a bitwise & with the exact bitmask of the bitfield.
For older GCC, we must disable -Wconversion for GCC 4 or 5 in such
cases.

The bitwise negation operator appears to promote short integers
to a wider type, and hence we must add explicit truncation casts
around them. Microsoft Visual C does not allow a static_cast to
truncate a constant, such as static_cast<byte>(1) truncating int.
Hence, we will use the constructor-style cast byte(~1) for such cases.

This has been tested at least with GCC 4.8.5, 5.4.0, 7.4.0, 9.2.1, 10.0.0,
clang 9.0.1, 10.0.0, and MSVC 14.22.27905 (Microsoft Visual Studio 2019)
on 64-bit and 32-bit targets (IA-32, AMD64, POWER 8, POWER 9, ARMv8).
2020-03-12 19:46:41 +02:00
Marko Mäkelä
d82ac8d374 MDEV-21907: Fix some -Wconversion outside InnoDB
Some .c and .cc files are compiled as part of Mariabackup.
Enabling -Wconversion for InnoDB would also enable it for
Mariabackup. The .h files are being included during
InnoDB or Mariabackup compilation.

Notably, GCC 5 (but not GCC 4 or 6 or later versions)
would report -Wconversion for x|=y when the type is
unsigned char. So, we will either write x=(uchar)(x|y)
or disable the -Wconversion warning for GCC 5.

bitmap_set_bit(), bitmap_flip_bit(), bitmap_clear_bit(), bitmap_is_set():
Always implement as inline functions.
2020-03-12 19:44:52 +02:00
Marko Mäkelä
c7920fa8ff MDEV-16264: Eliminate unsafe os_aio_userdata_t type cast 2020-03-12 19:43:45 +02:00
Marko Mäkelä
8be3794b42 MDEV-21924 Clean up InnoDB GIS record comparison
The extension of the record comparison functions for SPATIAL INDEX in
mysql/mysql-server@b66ad511b6
was suboptimal for multiple reasons:

Some functions used unnecessary temporary variables of the int type,
instead of the more appropriate size_t, causing type mismatch.

Many functions unnecessarily required rec_get_offsets() to be
computed, or a parameter for length, although the size of the
minimum bounding rectangle (MBR) is hard-coded as
SPDIMS * 2 * sizeof(double), or 32 bytes.

In InnoDB SPATIAL INDEX records, there always is a 32-byte key
followed by either a 4-byte child page number or the PRIMARY KEY value.

The length parameters were not properly validated.
The function cmp_geometry_field() was making an incorrect attempt
at checking that the lengths are at least sizeof(double) (8 bytes),
even though the function is accessing up to 32 bytes in both MBR.

Functions that are called from only one compilation unit are defined
in another compilation unit, making the code harder to follow and
potentially slower to execute.

cmp_dtuple_rec_with_gis(): FIXME: Correct the debug assertion
and possibly the function TABLE_SHARE::init_from_binary_frm_image()
or related code, which causes an unexpected length of
DATA_MBR_LEN + 2 bytes to be passed to this function.
2020-03-12 18:13:53 +02:00
Marko Mäkelä
0d76777872 MDEV-13362: Fix -Wset-but-unused 2020-03-12 18:13:47 +02:00
Julius Goryavsky
28fabc86db MDEV-13362: implement --require_secure_transport option
Currently, if a user wants to require TLS for every connection made
over the network, then every user account on the system needs to be
created with "REQUIRE SSL" or one of the other TLS options.

Implementing a require_secure_transport system varuable (which,
in particular, can be set using the --require_secure_transport=ON
command line option) in the MariaDB Server would make it a lot
easier to require TLS (or other secure transport) system-wide.

This patch implements this new system variable, adds the ability
to set it with SQL statements, from the command line and from the
configuration file, and also contains improvements for mtr that allow
the user to establish non-secure TCP/IP connections (for example,
to verify the operation of the new option).
2020-03-12 11:46:08 +01:00
Oleksandr Byelkin
fad47df995 Merge branch '10.4' into 10.5 2020-03-11 17:52:49 +01:00
Oleksandr Byelkin
b7362d5fbc Merge branch '10.3' into 10.4 2020-03-11 14:28:24 +01:00
Oleksandr Byelkin
3c9bc0ce19 Merge branch '10.2' into 10.3 2020-03-11 14:05:41 +01:00
Oleksandr Byelkin
b8c0e49670 Merge commit '10.3' into 10.4 2020-03-11 13:27:10 +01:00
Vladislav Vaintroub
9d7ed94f6a CMake cleanup - simplify create_initial_db.cmake
Also make initial_database optional target on non-Windows, as sometimes
it can be quite handy to bootstrap without MTR.
2020-03-11 10:58:53 +01:00
Aleksey Midenkov
8fa1b6bb88 MDEV-15724 - Possible crash in parser
Parser: uninitialized Lex->create_last_non_select_table under
mysql_unpack_partition() fix.

Tested with main, parts suites.
2020-03-11 08:40:37 +02:00
Marko Mäkelä
574d8b2940 MDEV-21907: Fix most clang -Wconversion in InnoDB
Declare innodb_purge_threads as 4-byte integer (UINT)
instead of 4-or-8-byte (ULONG) and adjust the documentation string.
2020-03-11 08:29:48 +02:00
Marko Mäkelä
6ec3682371 MDEV-21743: Re-record --big-test results 2020-03-11 08:29:48 +02:00
Marko Mäkelä
4f4fccecb2 Fix perfschema.statement_program_concurrency 2020-03-11 08:29:48 +02:00
Alexander Barkov
0e04beb28f Recoding new suite/perfschema/r/start_server_low_digest_sql_length.result
Fixing a test failure introduced by MDEV-21743
2020-03-11 00:35:30 +04:00
Alexander Barkov
a1e330de5a MDEV-21743 Split up SUPER privilege to smaller privileges 2020-03-10 23:49:47 +04:00
Oleksandr Byelkin
91ba789aaf fix for VALGRIND build 2020-03-10 20:36:18 +01:00
Oleksandr Byelkin
7676208ce6 Fixed problem of exiting over 32 bit on windows. 2020-03-10 20:12:30 +01:00
Sergei Golubchik
43460839dd MDEV-21795 10.5 with bundled pcre2 and embedded server could not be built
partially reverting 961413d26f
2020-03-10 19:24:24 +01:00
Sergei Golubchik
7180afa094 fix perfschema for pool-of-threads 2020-03-10 19:24:24 +01:00
Sergei Golubchik
a9b8131d99 cleanup: remove source files that aren't used 2020-03-10 19:24:23 +01:00
Sergei Golubchik
cbede21d0d cleanup: pass trxid by value 2020-03-10 19:24:23 +01:00
Sergei Golubchik
211421d5cc cleanup: remove unused argument 2020-03-10 19:24:23 +01:00
Sergei Golubchik
c1c5222cae cleanup: PSI key is *always* the first argument 2020-03-10 19:24:23 +01:00
Sergei Golubchik
7af733a5a2 perfschema compilation, test and misc fixes 2020-03-10 19:24:23 +01:00
Sergei Golubchik
81cffda2e6 perfschema transaction instrumentation related changes 2020-03-10 19:24:23 +01:00