Commit graph

557 commits

Author SHA1 Message Date
Monty
546e7aa96f MDEV-8203 Assert in Query_log_event::do_apply_event()
This happens because the master writes a table_map event to the binary log, but no row event.
The slave has a check that there should always be a row event if there was a table_map event, which
causes a crash.

Fixed by remembering in the cache what kind of events are logged
and ignore cached statements which is just a table map event.
2017-04-07 15:58:17 +04:00
Marko Mäkelä
89d80c1b0b Fix many -Wconversion warnings.
Define my_thread_id as an unsigned type, to avoid mismatch with
ulonglong.  Change some parameters to this type.

Use size_t in a few more places.

Declare many flag constants as unsigned to avoid sign mismatch
when shifting bits or applying the unary ~ operator.

When applying the unary ~ operator to enum constants, explictly
cast the result to an unsigned type, because enum constants can
be treated as signed.

In InnoDB, change the source code line number parameters from
ulint to unsigned type. Also, make some InnoDB functions return
a narrower type (unsigned or uint32_t instead of ulint;
bool instead of ibool).
2017-03-07 19:07:27 +02:00
Monty
d75d8631ed [MDEV-10570] Add Flashback support
==== Description ====

Flashback can rollback the instances/databases/tables to an old snapshot.
It's implement on Server-Level by full image format binary logs (--binlog-row-image=FULL), so it supports all engines.
Currently, it’s a feature inside mysqlbinlog tool (with --flashback arguments).

Because the flashback binlog events will store in the memory, you should check if there is enough memory in your machine.

==== New Arguments to mysqlbinlog ====

--flashback (-B)
It will let mysqlbinlog to work on FLASHBACK mode.

==== New Arguments to mysqld ====

--flashback

Setup the server to use flashback. This enables binary log in row mode
and will enable extra logging for DDL's needed by flashback feature

==== Example ====

I have a table "t" in database "test", we can compare the output with "--flashback" and without.

#client/mysqlbinlog /data/mysqldata_10.0/binlog/mysql-bin.000001 -vv -d test -T t --start-datetime="2013-03-27 14:54:00" > /tmp/1.sql
#client/mysqlbinlog /data/mysqldata_10.0/binlog/mysql-bin.000001 -vv -d test -T t --start-datetime="2013-03-27 14:54:00" -B > /tmp/2.sql

Then, importing the output flashback file (/tmp/2.log), it can flashback your database/table to the special time (--start-datetime).
And if you know the exact postion, "--start-postion" is also works, mysqlbinlog will output the flashback logs that can flashback to "--start-postion" position.

==== Implement ====

1. As we know, if binlog_format is ROW (binlog-row-image=FULL in 10.1 and later), all columns value are store in the row event, so we can get the data before mis-operation.

2. Just do following things:

  2.1 Change Event Type, INSERT->DELETE, DELETE->INSERT.
  For example:
    INSERT INTO t VALUES (...)  ---> DELETE FROM t WHERE ...
    DELETE FROM t ... ---> INSERT INTO t VALUES (...)

  2.2 For Update_Event, swapping the SET part and WHERE part.
  For example:
    UPDATE t SET cols1 = vals1 WHERE cols2 = vals2
    --->
    UPDATE t SET cols2 = vals2 WHERE cols1 = vals1

  2.3 For Multi-Rows Event, reverse the rows sequence, from the last row to the first row.
  For example:
    DELETE FROM t WHERE id=1; DELETE FROM t WHERE id=2; ...; DELETE FROM t WHERE id=n;
    --->
    DELETE FROM t WHERE id=n; ...; DELETE FROM t WHERE id=2; DELETE FROM t WHERE id=1;

  2.4 Output those events from the last one to the first one which mis-operation happened.
  For example:
2017-01-20 15:33:28 +02:00
Sergei Golubchik
4a5d25c338 Merge branch '10.1' into 10.2 2016-12-29 13:23:18 +01:00
Sergei Golubchik
b5aa0f437f MDEV-11319 mysqlbinlog crashes or fails with out of memory while reading some encrypted binlogs
support encrypted binlogs. Not decryption, but at least recognizing
that event are encrypted and prining them as such
2016-12-06 09:45:50 +01:00
vinchen
0e380c3bfe two fix:
1.Avoid overflowing buffers in case of corrupt events
2.Check the compressed algorithm.
2016-10-29 21:59:20 +08:00
Kristian Nielsen
c06bc66816 MDEV-11065: Compressed binary log
Minor review comments/changes:

 - A bunch of style-fixes.

 - Change macros to static inline functions.

 - Update check_event_type() with compressed event types.

 - Small .result file update.
2016-10-20 18:00:59 +02:00
vinchen
d4b2c9bb1a optimize the memory allocation for compressed binlog event 2016-10-19 20:20:47 +02:00
vinchen
640051e06a Binlog compressed
Add some event types for the compressed event, there are:
     QUERY_COMPRESSED_EVENT,
     WRITE_ROWS_COMPRESSED_EVENT_V1,
     UPDATE_ROWS_COMPRESSED_EVENT_V1,
     DELETE_POWS_COMPRESSED_EVENT_V1,
     WRITE_ROWS_COMPRESSED_EVENT,
     UPDATE_ROWS_COMPRESSED_EVENT,
     DELETE_POWS_COMPRESSED_EVENT.
These events inheritance the uncompressed editor events. One of their constructor functions and write
function have been overridden for uncompressing and compressing. Anything but this is totally the same.

On slave, The IO thread will uncompress and convert them When it receiving the events from the master.
So the SQL and worker threads can be stay unchanged.

Now we use zlib as compress algorithm. It maybe support other algorithm in the future.
2016-10-19 20:20:35 +02:00
Monty
7b96416f3c Use sql_mode_t for sql_mode.
This fixed several cases where we where using just ulong for sql_mode
2016-10-05 01:11:08 +03:00
Michael Widenius
db7edfed17 MDEV-7563 Support CHECK constraint as in (or close to) SQL Standard
MDEV-10134 Add full support for DEFAULT

- Added support for using tables with MySQL 5.7 virtual fields,
  including MySQL 5.7 syntax
- Better error messages also for old cases
- CREATE ... SELECT now also updates timestamp columns
- Blob can now have default values
- Added new system variable "check_constraint_checks", to turn of
  CHECK constraint checking if needed.
- Removed some engine independent tests in suite vcol to only test myisam
- Moved some tests from 'include' to 't'. Should some day be done for all tests.
- FRM version increased to 11 if one uses virtual fields or constraints
- Changed to use a bitmap to check if a field has got a value, instead of
  setting HAS_EXPLICIT_VALUE bit in field flags
- Expressions can now be up to 65K in total
- Ensure we are not refering to uninitialized fields when handling virtual fields or defaults
- Changed check_vcol_func_processor() to return a bitmap of used types
- Had to change some functions that calculated cached value in fix_fields to do
  this in val() or getdate() instead.
- store_now_in_TIME() now takes a THD argument
- fill_record() now updates default values
- Add a lookahead for NOT NULL, to be able to handle DEFAULT 1+1 NOT NULL
- Automatically generate a name for constraints that doesn't have a name
- Added support for ALTER TABLE DROP CONSTRAINT
- Ensure that partition functions register virtual fields used. This fixes
  some bugs when using virtual fields in a partitioning function
2016-06-30 11:43:02 +02:00
Monty
78771189e7 Ignore MySQL 5.7 log events not relevant for MariaDB
- XA
- Transaction_context_event (used by MysQL group replication)
- View change event (used by MysQL group replication)
2015-11-10 13:46:57 +02:00
Monty
93d1e5ce0b table->write_set was changed if binary logging was used, which caused the
changes in query execution plans.
Fixed by introducing table->rpl_write_set which holds which columns should
be stored in the binary log.

Other things:
- Removed some not needed references to read_set and write_set to make
  code really changing read_set and write_set easier to read
  (in opt_range.cc)
- Added error handling of failed unpack_current_row()
- Added missing call to mark_columns_needed_for_insert() for DELAYED INSERT
- Removed not used functions in_read_set() and in_write_set()
- In rpl_record.cc, removed not used variable error
2015-11-10 13:46:56 +02:00
Michael Widenius
c3ebd7867d Remove THD argment from Log_event->net_send() and Protocol::pack_info() as
THD is already available in Protocol
2015-10-22 17:00:58 +03:00
Sergei Golubchik
dfb74dea30 Merge branch '10.0' into 10.1 2015-10-12 00:37:58 +02:00
Monty
a69a6ddac8 MDEV-4487 Allow replication from MySQL 5.6+ when GTID is enabled on the master
MDEV-8685 MariaDB fails to decode Anonymous_GTID entries
MDEV-5705 Replication testing: 5.6->10.0

- Ignoring GTID events from MySQL 5.6+ (Allows replication from MySQL 5.6+ with GTID enabled)
- Added ignorable events from MySQL 5.6
- mysqlbinlog now writes information about GTID and ignorable events.
- Added more information in error message when replication stops because of wrong information in binary log.
- Fixed wrong test when write_on_release() should flush cache.
2015-10-08 10:45:09 +03:00
Oleksandr Byelkin
0ce0b88080 MDEV-8450: PATCH] Wrong macro expansion in Query_cache::send_result_to_client()
Expression in macro protected by ()
2015-09-06 22:26:33 +02:00
Sergei Golubchik
b85a00161e MDEV-8264 encryption for binlog
* Start_encryption_log_event
* --encrypt-binlog command line option

based on google patches.
2015-09-04 10:33:55 +02:00
Sergei Golubchik
41d68cabee cleanup: Log_event::write() and MYSQL_BIN_LOG::write_cache()
Introduce Log_event_writer() that encapsulates
writing data to an IO_CACHE with automatic checksum calculation.

Now all events properly checksum themselves as needed.

Use Log_event_writer in MYSQL_BIN_LOG::write_cache() instead
of copy-pasting its logic all over.

Later Log_event_writer will also do encryption.
2015-09-04 10:33:55 +02:00
Sergei Golubchik
274a47a50b cleanup: remove Slave_log_event (unused since 2002) 2015-09-04 10:33:52 +02:00
Sergei Golubchik
2d2286faf3 cleanup: use enum_binlog_checksum_alg, not uint8
* fix unireg.h includes
* use enum_binlog_checksum_alg for binlog checksum variables,
  not uint8
2015-09-04 10:33:52 +02:00
Sergei Golubchik
7b54dec1c6 cleanup: comments 2015-09-04 10:33:52 +02:00
Sergei Golubchik
89e08bf300 cleanup: reformat
* reformat long lines
* remove useless 'const'
* remove unneeded do { ... } while(false} wrapper
* remove unneeded method
2015-09-04 10:33:51 +02:00
Sergei Golubchik
5018a6606f cleanup: Log_event::read_log_event()
There are three Log_event::read_log_event() methods:
1. read the event image from IO_CACHE into String
2. create Log_event from the in-memory event image
3. read the event image from IO_CACHE and create Log_event

The 3rd was reading event image into memory and invoking the 2nd to
create Log_event. Now the 3rd also uses the 1st to read the event image
from IO_CACHE into memory, instead of duplicating its functionality.
2015-09-04 10:33:50 +02:00
Sergey Vojtovich
31e365efae MDEV-8010 - Avoid sql_alloc() in Items (Patch )
Added mandatory thd parameter to Item (and all derivative classes) constructor.
Added thd parameter to all routines that may create items.
Also removed "current_thd" from Item::Item. This reduced number of
pthread_getspecific() calls from 290 to 177 per OLTP RO transaction.
2015-08-21 10:40:39 +04:00
Vicențiu Ciorbaru
d817267ae6 [MDEV-6877] Change replication event loop to account for empty events
When writing rows with a minimal row image, it is possible to receive
empty events. In that case m_curr_row and m_rows_end are the same,
however the event implies an insert into the table with the default
values associated for that table.
2015-06-30 13:17:09 +03:00
Vicențiu Ciorbaru
ca276729bc [MDEV-6877] Fixed Update_rows_log_event to correctly apply update log events
Update_rows_log_event must make use of the after image bitmap when
detecting which columns to update.
2015-06-30 13:17:09 +03:00
Vicențiu Ciorbaru
8bd5301a1e [MDEV-6877] Add binlog writing according to columns
The logging function now writes to the binlog according
to the bitmap columns implied by the binlog_row_image variable.
2015-06-30 13:17:09 +03:00
Vicențiu Ciorbaru
edff3f3fa9 [MDEV-6877] Update Update, Delete and Write row log event
The row events no longer require columns arguments.
2015-06-30 13:17:09 +03:00
Vicențiu Ciorbaru
a7d181a023 [MDEV-6877] Added a bitmap compare function for binlog_row_image
The function compares bitmaps according to the binlog_row_image variable
setting.
2015-06-30 13:17:09 +03:00
Kristian Nielsen
8672339328 MDEV-6676: Optimistic parallel replication
Adjust the configuration options, as discussed on the
maria-developers@ mailing list.

The option to hint a transaction to not be replicated in parallel is
now called @@skip_parallel_replication, consistent with
@@skip_replication.

And the --slave-parallel-mode is now simplified to have just one of
the following values:

  none
  minimal
  conservative
  optimistic
  aggressive

This reflects successively harder efforts to find opportunities to run
things in parallel on the slave. It allows to extend the server with
more automatic heuristics in the future without having to introduce a
new configuration option for each and every one.
2015-02-07 09:42:58 +01:00
Sergei Golubchik
4b21cd21fe Merge branch '10.0' into merge-wip 2015-01-31 21:48:47 +01:00
Sergei Golubchik
d9c01e4b4a 5.5 merge 2015-01-21 12:03:02 +01:00
Sergei Golubchik
a978bdda1e mysql-5.5.41 merge 2014-12-19 11:35:44 +01:00
Kristian Nielsen
db21fddc37 MDEV-6676: Optimistic parallel replication
Implement a new mode for parallel replication. In this mode, all transactions
are optimistically attempted applied in parallel. In case of conflicts, the
offending transaction is rolled back and retried later non-parallel.

This is an early-release patch to facilitate testing, more changes to user
interface / options will be expected. The new mode is not enabled by default.
2014-12-06 08:49:50 +01:00
Sergei Golubchik
f62c12b405 Merge 10.0.14 into 10.1 2014-10-15 12:59:13 +02:00
Sujatha Sivakumar
9296809134 Bug#19145698: READ OUT OF BOUNDS ISSUE
Problem:
========
In a master slave replication if a slave receives a
Start_log_event_v3 the payload is expected to be of fixed
size. If a payload which is smaller than the fixed size is
received it causes a read out of bounds issue.

Analysis:
========
According to documentation the fixed data part of
Start_log_event_v3 looks as shown below.

2 bytes: The binary log format version
50 bytes: The MySQL server's version
4 bytes: Timestamp in seconds when this event was created

Since the payload is expected to be of fixed size, therefore
ST_SERVER_VER_LEN (50) bytes are memcpy'ed into
server_version. But if a malicious master sends a shorter
payload it causes a read out of bounds issue.

Fix:
===
In Start_log_event_v3 event's constructor a check has been
added which expects the minimum payload length to be of size
common_header_len + ST_COMMON_HEADER_LEN_OFFSET bytes. If a
malicious packet of lesser length is received it will be
considered as an invalid event.

sql/log_event.cc:
  Added code changes to check the minimum packet length
  of Start_log_event_v3 should be > 56.
sql/log_event.h:
  Moved server_version from stack to heap and modified
  is_valid function for Start_log_event_v3.
2014-10-08 10:50:02 +05:30
Jan Lindström
df4dd593f2 MDEV-6247: Merge 10.0-galera to 10.1.
Merged lp:maria/maria-10.0-galera up to revision 3879.

Added a new functions to handler API to forcefully abort_transaction,
producing fake_trx_id, get_checkpoint and set_checkpoint for XA. These
were added for future possiblity to add more storage engines that
could use galera replication.
2014-08-26 15:43:46 +03:00
Kristian Nielsen
00467e136e MDEV-5799: Error messages written upon LOST EVENTS incident are corrupted
This is MySQL Bug#59123. The message string stored in an INCIDENT event was
not zero-terminated. This caused any following checksum bytes (if enabled on
the master) to be output to the error log as trailing garbage when the message
was printed to the error log.

Backport the patch from MySQL 5.6:

  revno: 2876.228.200
  revision-id: zhenxing.he@sun.com-20110111051323-w2xnzvcjn46x6h6u
  committer: He Zhenxing <zhenxing.he@sun.com>
  timestamp: Tue 2011-01-11 13:13:23 +0800
  message:
    BUG#59123 rpl_stm_binlog_max_cache_size fails sporadically with found warnings

Also add a test case.
2014-06-25 13:08:30 +02:00
Sergei Golubchik
edf1fbd25b MDEV-6153 Trivial Lintian errors in MariaDB sources: spelling errors and wrong executable bits 2014-05-13 11:53:30 +02:00
Sergei Golubchik
d3e2e1243b 5.5 merge 2014-05-09 12:35:11 +02:00
Sergei Golubchik
302e4776e0 comments 2014-05-01 14:05:44 +02:00
Sergei Golubchik
218280bcd7 make append_query_string() more usable:
simplify the prototype and move it to sql_string.h
2014-03-26 09:42:33 +01:00
unknown
e7704bfdd6 RBR triggers compiled-out with ifdefs in 10.0 2014-03-19 11:35:32 +02:00
unknown
af3180ab6f MDEV-9095: Executing triggers on slave in row-based replication 2014-03-19 11:00:56 +02:00
unknown
5ec49e6452 Merge MDEV-5754, MDEV-5769, and MDEV-5764 into 10.0 2014-03-04 14:32:42 +01:00
unknown
b5b8210849 MDEV-5754: MySQL 5.5 slaves cannot replicate from MariaDB 10.0
The problem was when a GTID event was part of a group commit, and so contained
a commit id. The code that replaces GTID with a BEGIN event for old slaves did
not correctly handle this case.

Fix the code so that the GTID with commit id can also be properly replaced
with a BEGIN query event. The extra two bytes are in the BEGIN event replaced
with a dummy, empty time zone string.
2014-03-04 13:10:14 +01:00
Sergei Golubchik
cb67dcb618 mysql-5.5.37 selective merge 2014-03-27 22:26:58 +01:00
Sergei Golubchik
0dc23679c8 10.0-base merge 2014-02-26 15:28:07 +01:00
Sergei Golubchik
0b9a0a3517 5.5 merge 2014-02-25 16:04:35 +01:00