Commit graph

2271 commits

Author SHA1 Message Date
Marko Mäkelä
1eb2d8f6e8 Merge 10.2 into 10.3 2018-08-16 08:54:58 +03:00
Vladislav Vaintroub
922e7badfc MDEV-16791 mariabackup : Support DDL commands during backup 2018-08-14 15:11:13 +01:00
Vladislav Vaintroub
1faaaa9718 MDEV-15680 xb_aws_key_management fails in buildbot.
aws_key_management needs current directory to be datadir during
initalization, it scans current directory for encrypted keys.

Fix is to ensure, that plugin initialization in mariabackup happens
after the call to my_setwd(mysql_real_data_home).
2018-08-13 22:39:31 +01:00
Marko Mäkelä
f6d4f624eb MDEV-12041: innodb_encrypt_log key rotation
This will change the InnoDB encrypted redo log format only.
Unencrypted redo log will keep using the MariaDB 10.3 format.
In the new encrypted redo log format, 4 additional bytes will
be reserved in the redo log block trailer for storing the
encryption key version.

For performance reasons, the encryption key rotation
(checking if the latest encryption key version is being used)
is only done at log_checkpoint().

LOG_HEADER_FORMAT_CURRENT: Remove.

LOG_HEADER_FORMAT_ENC_10_4: The encrypted 10.4 format.

LOG_BLOCK_KEY: The encryption key version field.

LOG_BLOCK_TRL_SIZE: Remove.

log_t: Add accessors framing_size(), payload_size(), trailer_offset(),
to be used instead of referring to LOG_BLOCK_TRL_SIZE.

log_crypt_t: An operation passed to log_crypt().

log_crypt(): Perform decryption, encryption, or encryption with key
rotation. Return an error if key rotation at decryption fails.
On encryption, keep using the previous key if the rotation fails.
At startup, old-format encrypted redo log may be written before
the redo log is upgraded (rebuilt) to the latest format.

log_write_up_to(): Add the parameter rotate_key=false.

log_checkpoint(): Invoke log_write_up_to() with rotate_key=true.
2018-08-13 16:04:37 +03:00
Sergei Golubchik
0aa9b03393 Merge branch '10.2' into 10.3 2018-08-12 12:02:23 +02:00
Sergei Golubchik
675e7e7dcc remove obsolete checks for -fno-implicit-templates
we stopped using -fno-implicit-templates many years ago
2018-08-12 11:37:43 +02:00
Sergei Golubchik
ba1c05cc0d compiler warning
warning: suggest a space before ‘;’ or explicit braces around empty body in ‘for’ statement
2018-08-09 11:28:38 +02:00
Vladislav Vaintroub
3570ea9189 remove warning on Windows
Replace do-it-yourself version of strdup() with real strdup().
2018-08-05 00:36:59 +01:00
Marko Mäkelä
05459706f2 Merge 10.2 into 10.3 2018-08-03 15:57:23 +03:00
Marko Mäkelä
7b145fae13 mariabackup: Use snprintf() instead of sprintf() 2018-08-03 13:06:35 +03:00
Marko Mäkelä
814ae57daf Merge 10.1 into 10.2 2018-08-03 13:02:56 +03:00
Marko Mäkelä
769f6d2db7 Fix -Wclass-memaccess in WSREP,InnoDB,XtraDB 2018-08-03 12:21:13 +03:00
Marko Mäkelä
ef3070e997 Merge 10.1 into 10.2 2018-08-02 08:19:57 +03:00
Michael Widenius
0c745c743c Don't give warnings from perror or resolveip with safemalloc
- Added my_end(0) before exit
- Fixed typo in mysql_install_db
2018-07-29 10:56:11 +03:00
Alexander Barkov
e61568ee93 Merge remote-tracking branch 'origin/10.3' into 10.4 2018-07-03 14:02:05 +04:00
Sergei Golubchik
36e59752e7 Merge branch '10.2' into 10.3 2018-06-30 16:39:20 +02:00
Sergei Petrunia
f46acd4a3a Adopt Debian's fix-FTBFS-on-GNU-Hurd.patch.
- Took the original patch by Ondrej Sury;
- Applied a fix for a known problem in the patch:
   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=882062
- Fixed a few other issues
2018-06-29 14:00:00 +03:00
Vladislav Vaintroub
95ef8de891 mariabackup - rename backup-rocksdb option to rocksdb-backup
to avoid "using unique option prefix 'backup' is error-prone", there is
already --backup option there.
2018-06-22 23:30:26 +01:00
Vladislav Vaintroub
ecc4682672 MDEV-16519 : mariabackup should fail if MDL could not be acquired with lock-ddl-per-table
There is a tiny chance for race condition during MDL acquisition.

If table is renamed just prior to
SELECT 1 FROM <table_name> LIMIT 0

then this query would  fail, yet mariabackup --backup does not handle
it as fatal error and continues, only to fail later during file copy.

The fix is to die on error, of MDL lock query fails.
2018-06-22 15:24:09 +01:00
Marko Mäkelä
0121d5a790 Merge 10.2 into 10.3 2018-06-18 15:43:59 +03:00
Marko Mäkelä
b8514c94f6 MDEV-16496 Mariabackup: Implement --verbose option to instrument InnoDB log apply
srv_print_verbose_log: Introduce the value 2 to refer to
mariabackup --verbose.

recv_recover_page(), recv_parse_log_recs(): Add output for
mariabackup --verbose.
2018-06-15 16:14:12 +03:00
Marko Mäkelä
ff317fe08e Follow-up to MDEV-16367 mariabackup: error: failed to copy enough redo log
Commit dc9c555415 moved the final phase of
the redo log copying to the background thread. This would sometimes cause
too little redo log to be copied at the end of the backup. We would only
guarantee copying up to the latest redo log checkpoint. This would produce
a consistent backup, but it could refer to a too old point of time.

xtrabackup_copy_log(), xtrabackup_copy_logfile(): Add the parameter 'last'.

xtrabackup_backup_low(): Copy any remaining part of the log after the
backup threads have terminated.
2018-06-15 13:31:43 +03:00
Marko Mäkelä
a79b033b35 MDEV-16457 mariabackup 10.2+ should default to innodb_checksum_algorithm=crc32
Since MariaDB Server 10.2.2 (and MySQL 5.7), the default value of
innodb_checksum_algorithm is crc32 (CRC-32C), not the inefficient "innodb"
checksum. Change Mariabackup to use the same default, so that checksum
validation (when using the default algorithm on the server) will take less
time during mariabackup --backup. Also, mariabackup --prepare should be
a little faster, and the server should read backups faster, because the
page checksums would only be validated against CRC-32C.
2018-06-14 14:23:20 +03:00
Vladislav Vaintroub
aba2d7301f MDEV-13122 Backup myrocksdb with mariabackup. 2018-06-07 15:13:54 +01:00
Marko Mäkelä
62d21ddac1 Merge 10.2 into 10.3 2018-06-07 15:07:00 +03:00
Marko Mäkelä
dc9c555415 MDEV-16367 mariabackup: error: failed to copy enough redo log
log_copying_thread(): Keep copying redo log until the end has been
reached. (Previously, we would stop copying as soon as
the first batch of xtrabackup_copy_logfile() returned.)

log_copying: Remove. Use log_copying_running instead.

copy_logfile: Remove. Log copying will now only be invoked from
2 places: from xtrabackup_backup_func() for the initial batch,
and from log_copying_thread() until all of the log has been read.
Use the global variable metadata_to_lsn for determining if the
final part of the log is being copied.

xtrabackup_copy_log(): Add diagnostic messages for terminating
the copying. These messages should be dead code, because
log_group_read_log_seg() should be checking for the same.

xtrabackup_copy_logfile(): Correct the retrying logic.
If anything was successfully read, process the portion that
was read. On failure, let the caller close dst_log_file.

io_watching_thread(): Stop throttling during the last phase
of copying the log (metadata_to_lsn!=0). The final copying
of the log will now be performed in log_copying_thread().

stop_backup_threads(): Clean up the message about stopping
the log copying thread.

xtrabackup_backup_low(): Read metadata_to_lsn from the latest
checkpoint header page, even if it is the first page.
Let the log_copying_thread take care of copying all of
the redo log.
2018-06-07 14:29:35 +03:00
Marko Mäkelä
619c277a6c Mariabackup: Make some globals static 2018-06-07 14:11:55 +03:00
Sergei Golubchik
37659ef43b mysys: remove dead ME_xxx flags 2018-06-04 12:32:22 +02:00
Yuqi
ee5124d714 Make MariaDB CRC32-lib platform independence ()
Make mariadb crc32 lib platform independent

It looks strange that someone can make use of 2 crc libraries
(Power64 or AArch64) at the same time.

The patch sets macros 'CRC32_LIBRARY' to make platform independence as an optional crc32 library.

Change-Id: I68bbf73cafb6a12f7fb105ad57d117b114a8c4af
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2018-06-01 14:06:01 +04:00
Sergei Golubchik
4ec8598c1d Merge branch 'github/10.2' into 10.3 2018-05-22 11:47:09 +02:00
Sergei Golubchik
3a7d7e180a cleanup: create_temp_file()
simplify. move common code inside, specify common flags inside,
rewrite dead code (`if (mode & O_TEMPORARY)` on Linux, where
`O_TEMPORARY` is always 0) to actually do something.
2018-05-21 16:34:10 +00:00
Vladislav Vaintroub
207e5ba316 mariabackup : Fix race condition when killing query waiting for MDL
Itcan happen that the connection is already gone during the window
between quering I_S.PROCESSLIST and KILL QUERY.

Fix is to tolerate ER_NO_SUCH_THREAD returned from KILL QUERY.

Add small improvement in message "Killing MDL query " to actually output
the query.
Also do not try to kill queries that are already in Killed state.
2018-05-19 17:04:47 +00:00
Sergei Golubchik
c9717dc019 Merge branch '10.2' into 10.3 2018-05-11 13:15:10 +02:00
Sergei Golubchik
9b1824dcd2 Merge branch '10.1' into 10.2 2018-05-10 13:01:42 +02:00
Vladislav Vaintroub
f98496da96 MDEV-16105: Mariabackup does not support SSL
The reason is the missing HAVE_OPENSSL define for mariabackup.
2018-05-08 19:52:08 +00:00
Daniel Black
c28be510d1 Power8: use C implementation of crc32 instead of ASM
Compiles to same vector code, just a bit simplier.
vec_crc32.c is now identical to upstream
(https://github.com/antonblanchard/crc32-vpmsum/).

C code by Rogerio Alves <rogealve@br.ibm.com>

This implemention has been tested on big endian too.

Signed-off-by: Daniel Black <daniel@linux.ibm.com>
2018-05-07 09:26:12 +10:00
Marko Mäkelä
8bbcc0d505 MDEV-12218: Put back mariabackup --innodb-flush-method
Implement innodb_flush_method as an enum parameter in Mariabackup,
instead of ignoring the option and hard-wiring it to a default value.

xb0xb.h: Remove. Only xtrabackup.cc refers to the enum parameters.

innodb_flush_method_names[], innodb_flush_method_typelib[]:
Define as non-static, so that mariabackup can share the definitions.

srv_file_flush_method: Change the type to ulong, to match the
assignment in init_one_value() and handle_options() in mariabackup.
2018-04-30 18:22:52 +03:00
Marko Mäkelä
baa5a43d8c MDEV-16045: Replace log_group_t with log_t::files
There is only one log_sys and only one log_sys.log.

log_t::files::create(): Replaces log_init().

log_t::files::close(): Replaces log_group_close(), log_group_close_all().

fil_close_log_files(): if (free) log_sys.log_close();
The callers that passed free=true used to call log_group_close_all().

log_header_read(): Replaces log_group_header_read().

log_t::files::file_header_bufs_ptr: Use a single allocation.

log_t::files::file_header_bufs[]: Statically allocate the pointers.

log_t::files::set_fields(): Replaces log_group_set_fields().

log_t::files::calc_lsn_offset(): Replaces log_group_calc_lsn_offset().
Simplify the computation by using fewer variables.

log_t::files::read_log_seg(): Replaces log_group_read_log_seg().

log_sys_t::complete_checkpoint(): Replaces log_io_complete_checkpoint().

fil_aio_wait(): Move the logic from log_io_complete().
2018-04-29 09:46:24 +03:00
Marko Mäkelä
d73a898d64 MDEV-16045: Allocate log_sys statically
There is only one redo log subsystem in InnoDB. Allocate the object
statically, to avoid unnecessary dereferencing of the pointer.

log_t::create(): Renamed from log_sys_init().

log_t::close(): Renamed from log_shutdown().

log_t::checkpoint_buf_ptr: Remove. Allocate log_t::checkpoint_buf
statically.
2018-04-29 09:46:24 +03:00
Marko Mäkelä
715e4f4320 MDEV-12218 Clean up InnoDB parameter validation
Bind more InnoDB parameters directly to MYSQL_SYSVAR and
remove "shadow variables".

innodb_change_buffering: Declare as ENUM, not STRING.

innodb_flush_method: Declare as ENUM, not STRING.

innodb_log_buffer_size: Bind directly to srv_log_buffer_size,
without rounding it to a multiple of innodb_page_size.

LOG_BUFFER_SIZE: Remove.

SysTablespace::normalize_size(): Renamed from normalize().

innodb_init_params(): A new function to initialize and validate
InnoDB startup parameters.

innodb_init(): Renamed from innobase_init(). Invoke innodb_init_params()
before actually trying to start up InnoDB.

srv_start(bool): Renamed from innobase_start_or_create_for_mysql().
Added the input parameter create_new_db.

SRV_ALL_O_DIRECT_FSYNC: Define only for _WIN32.

xb_normalize_init_values(): Merge to innodb_init_param().
2018-04-29 09:41:42 +03:00
Marko Mäkelä
9ed2b2b2b8 Do not divide or multiply by srv_page_size
Instead, shift by srv_page_size_shift.
2018-04-28 20:52:22 +03:00
Marko Mäkelä
a90100d756 Replace univ_page_size and UNIV_PAGE_SIZE
Try to use one variable (srv_page_size) for innodb_page_size.

Also, replace UNIV_PAGE_SIZE_SHIFT with srv_page_size_shift.
2018-04-28 20:45:45 +03:00
Monty
2ccd6716fc Fix a lot of compiler warnings found by -Wunused 2018-04-26 17:35:12 +03:00
Marko Mäkelä
7396dfcca7 Merge 10.2 into 10.3 2018-04-24 20:59:57 +03:00
Marko Mäkelä
4cd7979c56 Merge 10.1 into 10.2 2018-04-24 09:39:45 +03:00
Marko Mäkelä
5b79303b40 MDEV-15988 Crash in ./mtr mariabackup.undo_space_id
xb_assign_undo_space_start(): Correctly pass the length of
the buffer, so that the file name will not be truncated.
2018-04-24 09:19:34 +03:00
Marko Mäkelä
6c64101bf0 MDEV-12266 follow-up fix to Mariabackup
xtrabackup_apply_delta(): Refer to fil_system.sys_space directly.
2018-04-23 13:14:28 +03:00
Marko Mäkelä
c6ba758d1d Merge 10.2 into 10.3 2018-04-23 09:49:58 +03:00
Marko Mäkelä
ea94717983 Merge 10.1 into 10.2 2018-04-21 11:58:32 +03:00
Michael Widenius
ddc5764303 Remove compiler warnings
- Remove unused variables
- Mark variables unused
- Fix wrong types
- Add no-strict-aliasing to BUILD scripts
2018-04-16 20:16:43 +03:00