Commit graph

36 commits

Author SHA1 Message Date
Andrei Elkin
6112a0f93d MDEV-16372 ER_BASE64_DECODE_ERROR upon replaying binary log via mysqlbinlog --verbose
(This commit is exclusively for 10.2 branch. Do not merge it to 10.3)

In case of a pattern of non-STMT_END-marked Rows-log-event (A) followed by
a STMT_END marked one (B) mysqlbinlog mixes up the base64 encoded rows events
with their pseudo sql representation produced by the verbose option:
      BINLOG '
        base64 encoded data for A
        ### verbose section for A
        base64 encoded data for B
        ### verbose section for B
      '/*!*/;
In effect the produced BINLOG '...' query is not valid and is rejected with the error.
Examples of this way malformed BINLOG could have been found in binlog_row_annotate.result
that gets corrected with the patch.

The issue is fixed with introduction an auxiliary IO_CACHE to hold on the verbose
comments until the terminal STMT_END event is found. The new cache is emptied
out after two pre-existing ones are done at that time.
The correctly produced output now for the above case is as the following:
      BINLOG '
        base64 encoded data for A
        base64 encoded data for B
      '/*!*/;
        ### verbose section for A
        ### verbose section for B

Thanks to Alexey Midenkov for the problem recognition and attempt to tackle,
and to Venkatesh Duggirala who produced a patch for the upstream whose
idea is exploited here, as well as to MDEV-23077 reporter LukeXwang who
also contributed a piece of a patch aiming at this issue.
2020-08-31 18:37:44 +03:00
Andrei Elkin
a19cb3884f MDEV-23511 shutdown_server 10 times out, causing server kill at shutdown
Shutdown of mtr tests may be too impatient, esp on CI environment where
10 seconds of `arg` of `shutdown_server arg` may not be enough for the clean
shutdown to complete.

This is fixed to remove explicit non-zero timeout argument to
`shutdown_server` from all mtr tests. mysqltest computes 60 seconds default
value for the timeout for the argless `shutdown_server` command.
This policy is additionally ensured with a compile time assert.
2020-08-21 14:48:53 +03:00
Sujatha
450a5b33a2 MDEV-22451: SIGSEGV in __memmove_avx_unaligned_erms/memcpy from _my_b_write on CREATE after RESET MASTER
Merge branch '10.1' into 10.2
2020-05-20 20:49:04 +05:30
Sujatha
836d708997 MDEV-22451: SIGSEGV in __memmove_avx_unaligned_erms/memcpy from _my_b_write on CREATE after RESET MASTER
Analysis:
========
RESET MASTER TO # command deletes all binary log files listed in the index
file, resets the binary log index file to be empty, and creates a new binary
log with number #. When the user provided binary log number is greater than
the max allowed value '2147483647' server fails to generate a new binary log.
The RESET MASTER statement marks the binlog closure status as
'LOG_CLOSE_TO_BE_OPENED' and exits. Statements which follow RESET MASTER
try to write to binary log they find the log_state != LOG_CLOSED and
proceed to write to binary log cache and it results in crash.

Fix:
===
During MYSQL_BIN_LOG open, if generation of new binary log name fails then the
"log_state" needs to be marked as "LOG_CLOSED". With this further statements
will find binary log as closed and they will skip writing to the binary log.
2020-05-20 17:42:28 +05:30
Alice Sherepa
ad6e421bd2 MDEV-21360 restore debud_dbug through a session variable instead of '-d,..' 2020-03-23 10:57:21 +01:00
Alice Sherepa
90e7e6783b MDEV-21360 save/restore debud_dbug instead of total reset at the end of the test 2020-01-21 11:22:47 +01:00
Alice Sherepa
10a5e1eccb MDEV-21360 save/restore debud_dbug instead of total reset at the end of the test 2020-01-21 11:22:47 +01:00
Alice Sherepa
b7fb30e930 MDEV-21360 global debug_dbug pre-test value restoration issues 2020-01-15 18:06:24 +01:00
Alice Sherepa
451573fab1 MDEV-21360 debug_dbug pre-test value restoration issues 2020-01-15 18:06:24 +01:00
Marko Mäkelä
2350066e63 Merge 10.1 into 10.2 2019-11-12 14:36:37 +02:00
Sujatha
7df07c7666 MDEV-20953: binlog_encryption.rpl_corruption failed in buildbot due to wrong error code
Problem:
========
CURRENT_TEST: binlog_encryption.rpl_corruption

mysqltest: In included file "./include/wait_for_slave_io_error.inc":
...
At line 72: Slave stopped with wrong error code
**** Slave stopped with wrong error code: 1743 (expected 1595,1913) ****

Analysis:
========
The test emulates the corruption at the various stages of replication for
example in binlog file, in network and in relay log etc. It verifies that all
corruption cases are handled through appropriate error messages.

The test cases which emulate network failure expect following errors.
--ER_SLAVE_RELAY_LOG_WRITE_FAILURE (1595)
--ER_NETWORK_READ_EVENT_CHECKSUM_FAILURE (1743)

Ideally test should expect error codes as 1595 and 1743.
But the test actually waits on incorrect error code 1595,1913

Fix:
===
Added appropriate error code for 'ER_NETWORK_READ_EVENT_CHECKSUM_FAILURE'.
Replaced 1913 with 1743.
2019-11-12 16:31:08 +05:30
Sachin Setiya
27664ef29d MDEV-20574 Position of events reported by mysqlbinlog is wrong with encrypted binlogs, SHOW BINLOG EVENTS reports the correct one.
Analysis

Mysqlbinlog output for encrypted binary log
#Q> insert into tab1 values (3,'row 003')
#190912 17:36:35 server id 10221  end_log_pos 980 CRC32 0x53bcb3d3  Table_map: `test`.`tab1` mapped to number 19
# at 940
#190912 17:36:35 server id 10221  end_log_pos 1026 CRC32 0xf2ae5136     Write_rows: table id 19 flags: STMT_END_F

Here we can see Table_map_log_event ends at 980 but Next event starts at 940.
And the reason for that is we do not send START_ENCRYPTION_EVENT to the slave

Solution:-
Send Start_encryption_log_event as Ignorable_log_event to slave(mysqlbinlog),
So that mysqlbinlog can update its log_pos.
Since Slave can request multiple FORMAT_DESCRIPTION_EVENT while master does not
have so We only update slave master pos when master actually have the
FORMAT_DESCRIPTION_EVENT. Similar logic should be applied for START_ENCRYPTION_EVENT.

Also added the test case when new server reads the data from old server which
does not send START_ENCRYPTION_EVENT to slave.

Master Slave Upgrade Scenario.
When Slave is updated first, Slave will have extra logic of handling
START_ENCRYPTION_EVENT But master willnot be sending START_ENCRYPTION_EVENT.
So there will be no issue.
When Master is updated first, It will send  START_ENCRYPTION_EVENT to
slave , But slave will ignore this event in queue_event.
2019-10-08 14:35:34 +05:30
Alexander Barkov
f203245e9e Merge remote-tracking branch 'origin/10.1' into 10.2 2019-10-01 07:11:54 +04:00
Sujatha
9b80f9300d MDEV-20645: Replication consistency is broken as workers miss the error notification from an earlier failed group.
Analysis:
========
In general if there are three groups.
1 - Inserts 32 which fails due to local entry '32' on slave.
2 - Inserts 33
3 - Inserts 34

Each group considers itself as a waiter and it waits for prior group 'waitee'.
This is done in 'register_wait_for_prior_event_group_commit'. If there is no
other parallel group being scheduled then no waitee will be there.

Let us assume 3 groups are being scheduled in parallel.

3-> waits for 2-> waits for->1

'1' upon completion it checks is there any registered subsequent waiter. If
so it wakes up the subsequent waiter with its execution status. This execution
status is stored in wakeup_error.

If '1' failed then it sends corresponding wakeup_error to 2. Then '2' aborts
and it propagates error to '3'.  So all further commits are aborted.  This
mechanism works only when all transactions reach a stage where they are
waiting for their prior commit to complete.

In case of optimistic following scenario occurs.

1,2,3 are scheduled in parallel.

3 - Reaches group_commit_code waits for 2 to complete.
1 - errors out sets stop_on_error_sub_id=1.

When a group execution results in error its corresponding sub_id is set to
'stop_on_error_sub_id'. Any new groups queued for execution will check if
their sub_id is > stop_on_error_sub_id.  If it is true their execution will be
skipped as prior group execution failed.  'skip_event_group=1' will be set.
Since the execution of SQL thread is about to stop we just skip execution of
all the following event groups.  We still do all the normal waiting and wakeup
processing between the event groups as a simple way to ensure that everything
is stopped and cleaned up correctly.

Upon error '1' transaction checks for registered waiters. Since no one is
there it simply goes away.

2 - Starts the execution. It checks do I have a waitee.

Since wait_commit_sub_id == entry->last_committed_sub_id no waitee is set.

Secondly: 'entry->stop_on_error_sub_id' is set by '1'st execution.  Now
'handle_parallel_thread' code checks if the current group 'sub_id' is greater
than the 'sub_id' set within 'stop_on_error_sub_id'.

Since the above is true 'skip_event_group=true' is set.  Simply call
'wait_for_prior_commit' to wakeup all waiters.  Group '2' didn't had any
waitee and its execution is skipped.  Hence its wakeup_error=0.It sends a
positive wakeup signal to '3'. Which commits. This results in a missed
transaction. i.e 33 is missed and 34 is committed.

Fix:
===
When a worker learns that an earlier transaction execution has failed, and it
should not proceed for further execution, it should mark its own execution
status as failed so that it alerts its followers to abort as well.
2019-09-30 13:22:37 +05:30
Monty
9cba6c5aa3 Updated mtr files to support different compiled in options
This allows one to run the test suite even if any of the following
options are changed:
- character-set-server
- collation-server
- join-cache-level
- log-basename
- max-allowed-packet
- optimizer-switch
- query-cache-size and query-cache-type
- skip-name-resolve
- table-definition-cache
- table-open-cache
- Some innodb options
etc

Changes:
- Don't print out the value of system variables as one can't depend on
  them to being constants.
- Don't set global variables to 'default' as the default may not
  be the same as the test was started with if there was an additional
  option file. Instead save original value and reset it at end of test.
- Test that depends on the latin1 character set should include
  default_charset.inc or set the character set to latin1
- Test that depends on the original optimizer switch, should include
  default_optimizer_switch.inc
- Test that depends on the value of a specific system variable should
  set it in the test (like optimizer_use_condition_selectivity)
- Split subselect3.test into subselect3.test and subselect3.inc to
  make it easier to set and reset system variables.
- Added .opt files for test that required specfic options that could
  be changed by external configuration files.
- Fixed result files in rockdsb & tokudb that had not been updated for
  a while.
2019-09-01 19:17:35 +03:00
Andrei Elkin
2308b9afec MDEV-17098 DATE -> DATETIME replication conversion not working, even in ALL_NON_LOSSY mode
Opened up MYSQL_TYPE _DATETIME{,2} <-> _NEWDATE conversions for replication.
2018-10-16 12:56:33 +03:00
Marko Mäkelä
6f96ff7268 Allow tests to work with cmake -DPLUGIN_PARTITION=NO 2018-05-29 17:14:34 +03:00
Alexander Barkov
f5cb66fb97 Test cleanup for MDEV-15833 (uncommenting forgotten tests) 2018-04-10 14:27:18 +04:00
Alexander Barkov
4d9c5844b8 MDEV-15833 Row format replication between LONGBLOB and MEDIUMBLOB does not work for long values
The code in Type_handler_blob****::make_conversion_table_field()
erroneously assumed that row format replication uses
MYSQL_TYPE_TINYBLOB, MYSQL_TYPE_BLOB, MYSQL_TYPE_MEDIUMBLOB,
MYSQL_TYPE_LONGBLOB type codes to tranfer BLOB variations.

In fact, all BLOB variations use MYSQL_TYPE_BLOB as the type
code, while the BLOB packlength (1,2,3 or 4) it tranferred
in metadata.

The bug was introduced by  aee068085d
(MDEV-9238 Wrap create_virtual_tmp_table() into a class, split into different steps)
2018-04-10 13:15:57 +04:00
Sergei Golubchik
b2865a437f search_pattern_in_file.inc changes
1. Special mode to search in error logs: if SEARCH_RANGE is not set,
   the file is considered an error log and the search is performed
   since the last CURRENT_TEST: line
2. Number of matches is printed too. "FOUND 5 /foo/ in bar".
   Use greedy .* at the end of the pattern if number of matches
   isn't stable. If nothing is found it's still "NOT FOUND",
   not "FOUND 0".
3. SEARCH_ABORT specifies the prefix of the output.
   Can be "NOT FOUND" or "FOUND" as before,
   but also "FOUND 5 " if needed.
2017-03-31 19:28:58 +02:00
Sergei Golubchik
3d85292afd sporadic failures of rpl.rpl_binlog_errors
the test restarts the server, giving it 60 seconds to shutdown
and then killing it mercilessly.

make sure the server closes all MyISAM tables before shutdown,
as we cannot reliably expect it to make the deadline.
2017-02-14 20:43:41 +01:00
Sergei Golubchik
ef0db6bfd1 fix rpl_binlog_errors test not to leave dbug enabled 2017-02-13 18:12:04 +01:00
Sergei Golubchik
f3914d10b6 Merge branch 'bb-10.2-serg-merge' into 10.2 2017-02-11 09:45:34 +01:00
Sergei Golubchik
2195bb4e41 Merge branch '10.1' into 10.2 2017-02-10 17:01:45 +01:00
Nirbhay Choubey
8b2e642aa2 MDEV-7635: Update tests to adapt to the new default sql_mode 2017-02-10 06:30:42 -05:00
Igor Babaev
69114862f2 Adjusted more tests after the fix for mdev-9976. 2017-02-01 13:09:07 -08:00
Elena Stepanova
beeacd2287 Follow-up for MDEV-11065 - add tests for compressed+encrypted binlog 2017-01-23 01:06:15 +02:00
Vicențiu Ciorbaru
8e15768731 Merge branch '10.0' into 10.1 2017-01-16 03:18:14 +02:00
Sergei Golubchik
0d897c2ceb cleanup: binlog.binlog_row_annotate 2016-12-29 13:23:35 +01:00
Sergei Golubchik
4a5d25c338 Merge branch '10.1' into 10.2 2016-12-29 13:23:18 +01:00
Sergei Golubchik
2f20d297f8 Merge branch '10.0' into 10.1 2016-12-11 09:53:42 +01:00
Elena Stepanova
8e702bce66 MDEV-11504 binlog_encryption.encrypted_master_switch_to_unencrypted fails sporadically in buildbot
The reason is a simple race condition. Initially the test was meant to synchronize with master
before showing tables, but it turned out that the slave IO thread should fail by this point,
and synchronization was removed along with a server bugfix. Now added an intermediate sync
instead, to make sure that slave has replicated events before the point of failure
2016-12-08 17:05:01 +02:00
Sergei Golubchik
74d52ded72 fix binlog_encryption.binlog_incident test 2016-12-06 09:45:51 +01:00
Sergei Golubchik
b5aa0f437f MDEV-11319 mysqlbinlog crashes or fails with out of memory while reading some encrypted binlogs
support encrypted binlogs. Not decryption, but at least recognizing
that event are encrypted and prining them as such
2016-12-06 09:45:50 +01:00
Sergei Golubchik
952856c810 MDEV-11288 Server crashes in Binlog_crypt_data::init trying to feed encrypted log without decryption capabilities 2016-12-06 09:45:50 +01:00
Elena Stepanova
611f91605a MDEV-9038 Binlog encryption tests
- created binlog_encryption test suite and added it to the default list
- moved some tests from rpl, binlog and multisource suites to extra
  so that they could be re-used in different suites
- made minor changes in include files
2016-12-05 20:19:01 +02:00