Commit graph

420 commits

Author SHA1 Message Date
Vladislav Vaintroub
f926c5f4fa MDEV-16996 mariabackup --prepare does not use native AIO on Linux by default 2018-08-16 08:37:54 +01:00
Marko Mäkelä
8716bb3b72 After-merge fix: Restore DECLARE_THREAD to fix Windows build 2018-08-16 09:41:20 +03:00
Marko Mäkelä
1eb2d8f6e8 Merge 10.2 into 10.3 2018-08-16 08:54:58 +03:00
Vladislav Vaintroub
922e7badfc MDEV-16791 mariabackup : Support DDL commands during backup 2018-08-14 15:11:13 +01:00
Vladislav Vaintroub
1faaaa9718 MDEV-15680 xb_aws_key_management fails in buildbot.
aws_key_management needs current directory to be datadir during
initalization, it scans current directory for encrypted keys.

Fix is to ensure, that plugin initialization in mariabackup happens
after the call to my_setwd(mysql_real_data_home).
2018-08-13 22:39:31 +01:00
Sergei Golubchik
0aa9b03393 Merge branch '10.2' into 10.3 2018-08-12 12:02:23 +02:00
Sergei Golubchik
ba1c05cc0d compiler warning
warning: suggest a space before ‘;’ or explicit braces around empty body in ‘for’ statement
2018-08-09 11:28:38 +02:00
Vladislav Vaintroub
3570ea9189 remove warning on Windows
Replace do-it-yourself version of strdup() with real strdup().
2018-08-05 00:36:59 +01:00
Marko Mäkelä
05459706f2 Merge 10.2 into 10.3 2018-08-03 15:57:23 +03:00
Marko Mäkelä
7b145fae13 mariabackup: Use snprintf() instead of sprintf() 2018-08-03 13:06:35 +03:00
Marko Mäkelä
814ae57daf Merge 10.1 into 10.2 2018-08-03 13:02:56 +03:00
Marko Mäkelä
769f6d2db7 Fix -Wclass-memaccess in WSREP,InnoDB,XtraDB 2018-08-03 12:21:13 +03:00
Marko Mäkelä
ef3070e997 Merge 10.1 into 10.2 2018-08-02 08:19:57 +03:00
Sergei Golubchik
36e59752e7 Merge branch '10.2' into 10.3 2018-06-30 16:39:20 +02:00
Sergei Petrunia
f46acd4a3a Adopt Debian's fix-FTBFS-on-GNU-Hurd.patch.
- Took the original patch by Ondrej Sury;
- Applied a fix for a known problem in the patch:
   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=882062
- Fixed a few other issues
2018-06-29 14:00:00 +03:00
Vladislav Vaintroub
95ef8de891 mariabackup - rename backup-rocksdb option to rocksdb-backup
to avoid "using unique option prefix 'backup' is error-prone", there is
already --backup option there.
2018-06-22 23:30:26 +01:00
Vladislav Vaintroub
ecc4682672 MDEV-16519 : mariabackup should fail if MDL could not be acquired with lock-ddl-per-table
There is a tiny chance for race condition during MDL acquisition.

If table is renamed just prior to
SELECT 1 FROM <table_name> LIMIT 0

then this query would  fail, yet mariabackup --backup does not handle
it as fatal error and continues, only to fail later during file copy.

The fix is to die on error, of MDL lock query fails.
2018-06-22 15:24:09 +01:00
Marko Mäkelä
0121d5a790 Merge 10.2 into 10.3 2018-06-18 15:43:59 +03:00
Marko Mäkelä
b8514c94f6 MDEV-16496 Mariabackup: Implement --verbose option to instrument InnoDB log apply
srv_print_verbose_log: Introduce the value 2 to refer to
mariabackup --verbose.

recv_recover_page(), recv_parse_log_recs(): Add output for
mariabackup --verbose.
2018-06-15 16:14:12 +03:00
Marko Mäkelä
ff317fe08e Follow-up to MDEV-16367 mariabackup: error: failed to copy enough redo log
Commit dc9c555415 moved the final phase of
the redo log copying to the background thread. This would sometimes cause
too little redo log to be copied at the end of the backup. We would only
guarantee copying up to the latest redo log checkpoint. This would produce
a consistent backup, but it could refer to a too old point of time.

xtrabackup_copy_log(), xtrabackup_copy_logfile(): Add the parameter 'last'.

xtrabackup_backup_low(): Copy any remaining part of the log after the
backup threads have terminated.
2018-06-15 13:31:43 +03:00
Marko Mäkelä
a79b033b35 MDEV-16457 mariabackup 10.2+ should default to innodb_checksum_algorithm=crc32
Since MariaDB Server 10.2.2 (and MySQL 5.7), the default value of
innodb_checksum_algorithm is crc32 (CRC-32C), not the inefficient "innodb"
checksum. Change Mariabackup to use the same default, so that checksum
validation (when using the default algorithm on the server) will take less
time during mariabackup --backup. Also, mariabackup --prepare should be
a little faster, and the server should read backups faster, because the
page checksums would only be validated against CRC-32C.
2018-06-14 14:23:20 +03:00
Vladislav Vaintroub
aba2d7301f MDEV-13122 Backup myrocksdb with mariabackup. 2018-06-07 15:13:54 +01:00
Marko Mäkelä
62d21ddac1 Merge 10.2 into 10.3 2018-06-07 15:07:00 +03:00
Marko Mäkelä
dc9c555415 MDEV-16367 mariabackup: error: failed to copy enough redo log
log_copying_thread(): Keep copying redo log until the end has been
reached. (Previously, we would stop copying as soon as
the first batch of xtrabackup_copy_logfile() returned.)

log_copying: Remove. Use log_copying_running instead.

copy_logfile: Remove. Log copying will now only be invoked from
2 places: from xtrabackup_backup_func() for the initial batch,
and from log_copying_thread() until all of the log has been read.
Use the global variable metadata_to_lsn for determining if the
final part of the log is being copied.

xtrabackup_copy_log(): Add diagnostic messages for terminating
the copying. These messages should be dead code, because
log_group_read_log_seg() should be checking for the same.

xtrabackup_copy_logfile(): Correct the retrying logic.
If anything was successfully read, process the portion that
was read. On failure, let the caller close dst_log_file.

io_watching_thread(): Stop throttling during the last phase
of copying the log (metadata_to_lsn!=0). The final copying
of the log will now be performed in log_copying_thread().

stop_backup_threads(): Clean up the message about stopping
the log copying thread.

xtrabackup_backup_low(): Read metadata_to_lsn from the latest
checkpoint header page, even if it is the first page.
Let the log_copying_thread take care of copying all of
the redo log.
2018-06-07 14:29:35 +03:00
Marko Mäkelä
619c277a6c Mariabackup: Make some globals static 2018-06-07 14:11:55 +03:00
Sergei Golubchik
4ec8598c1d Merge branch 'github/10.2' into 10.3 2018-05-22 11:47:09 +02:00
Sergei Golubchik
3a7d7e180a cleanup: create_temp_file()
simplify. move common code inside, specify common flags inside,
rewrite dead code (`if (mode & O_TEMPORARY)` on Linux, where
`O_TEMPORARY` is always 0) to actually do something.
2018-05-21 16:34:10 +00:00
Vladislav Vaintroub
207e5ba316 mariabackup : Fix race condition when killing query waiting for MDL
Itcan happen that the connection is already gone during the window
between quering I_S.PROCESSLIST and KILL QUERY.

Fix is to tolerate ER_NO_SUCH_THREAD returned from KILL QUERY.

Add small improvement in message "Killing MDL query " to actually output
the query.
Also do not try to kill queries that are already in Killed state.
2018-05-19 17:04:47 +00:00
Sergei Golubchik
c9717dc019 Merge branch '10.2' into 10.3 2018-05-11 13:15:10 +02:00
Sergei Golubchik
9b1824dcd2 Merge branch '10.1' into 10.2 2018-05-10 13:01:42 +02:00
Vladislav Vaintroub
f98496da96 MDEV-16105: Mariabackup does not support SSL
The reason is the missing HAVE_OPENSSL define for mariabackup.
2018-05-08 19:52:08 +00:00
Marko Mäkelä
8bbcc0d505 MDEV-12218: Put back mariabackup --innodb-flush-method
Implement innodb_flush_method as an enum parameter in Mariabackup,
instead of ignoring the option and hard-wiring it to a default value.

xb0xb.h: Remove. Only xtrabackup.cc refers to the enum parameters.

innodb_flush_method_names[], innodb_flush_method_typelib[]:
Define as non-static, so that mariabackup can share the definitions.

srv_file_flush_method: Change the type to ulong, to match the
assignment in init_one_value() and handle_options() in mariabackup.
2018-04-30 18:22:52 +03:00
Marko Mäkelä
baa5a43d8c MDEV-16045: Replace log_group_t with log_t::files
There is only one log_sys and only one log_sys.log.

log_t::files::create(): Replaces log_init().

log_t::files::close(): Replaces log_group_close(), log_group_close_all().

fil_close_log_files(): if (free) log_sys.log_close();
The callers that passed free=true used to call log_group_close_all().

log_header_read(): Replaces log_group_header_read().

log_t::files::file_header_bufs_ptr: Use a single allocation.

log_t::files::file_header_bufs[]: Statically allocate the pointers.

log_t::files::set_fields(): Replaces log_group_set_fields().

log_t::files::calc_lsn_offset(): Replaces log_group_calc_lsn_offset().
Simplify the computation by using fewer variables.

log_t::files::read_log_seg(): Replaces log_group_read_log_seg().

log_sys_t::complete_checkpoint(): Replaces log_io_complete_checkpoint().

fil_aio_wait(): Move the logic from log_io_complete().
2018-04-29 09:46:24 +03:00
Marko Mäkelä
d73a898d64 MDEV-16045: Allocate log_sys statically
There is only one redo log subsystem in InnoDB. Allocate the object
statically, to avoid unnecessary dereferencing of the pointer.

log_t::create(): Renamed from log_sys_init().

log_t::close(): Renamed from log_shutdown().

log_t::checkpoint_buf_ptr: Remove. Allocate log_t::checkpoint_buf
statically.
2018-04-29 09:46:24 +03:00
Marko Mäkelä
715e4f4320 MDEV-12218 Clean up InnoDB parameter validation
Bind more InnoDB parameters directly to MYSQL_SYSVAR and
remove "shadow variables".

innodb_change_buffering: Declare as ENUM, not STRING.

innodb_flush_method: Declare as ENUM, not STRING.

innodb_log_buffer_size: Bind directly to srv_log_buffer_size,
without rounding it to a multiple of innodb_page_size.

LOG_BUFFER_SIZE: Remove.

SysTablespace::normalize_size(): Renamed from normalize().

innodb_init_params(): A new function to initialize and validate
InnoDB startup parameters.

innodb_init(): Renamed from innobase_init(). Invoke innodb_init_params()
before actually trying to start up InnoDB.

srv_start(bool): Renamed from innobase_start_or_create_for_mysql().
Added the input parameter create_new_db.

SRV_ALL_O_DIRECT_FSYNC: Define only for _WIN32.

xb_normalize_init_values(): Merge to innodb_init_param().
2018-04-29 09:41:42 +03:00
Marko Mäkelä
9ed2b2b2b8 Do not divide or multiply by srv_page_size
Instead, shift by srv_page_size_shift.
2018-04-28 20:52:22 +03:00
Marko Mäkelä
a90100d756 Replace univ_page_size and UNIV_PAGE_SIZE
Try to use one variable (srv_page_size) for innodb_page_size.

Also, replace UNIV_PAGE_SIZE_SHIFT with srv_page_size_shift.
2018-04-28 20:45:45 +03:00
Monty
2ccd6716fc Fix a lot of compiler warnings found by -Wunused 2018-04-26 17:35:12 +03:00
Marko Mäkelä
7396dfcca7 Merge 10.2 into 10.3 2018-04-24 20:59:57 +03:00
Marko Mäkelä
4cd7979c56 Merge 10.1 into 10.2 2018-04-24 09:39:45 +03:00
Marko Mäkelä
5b79303b40 MDEV-15988 Crash in ./mtr mariabackup.undo_space_id
xb_assign_undo_space_start(): Correctly pass the length of
the buffer, so that the file name will not be truncated.
2018-04-24 09:19:34 +03:00
Marko Mäkelä
6c64101bf0 MDEV-12266 follow-up fix to Mariabackup
xtrabackup_apply_delta(): Refer to fil_system.sys_space directly.
2018-04-23 13:14:28 +03:00
Marko Mäkelä
c6ba758d1d Merge 10.2 into 10.3 2018-04-23 09:49:58 +03:00
Marko Mäkelä
ea94717983 Merge 10.1 into 10.2 2018-04-21 11:58:32 +03:00
Michael Widenius
ddc5764303 Remove compiler warnings
- Remove unused variables
- Mark variables unused
- Fix wrong types
- Add no-strict-aliasing to BUILD scripts
2018-04-16 20:16:43 +03:00
Vladislav Vaintroub
c2dc72c0c3 MDEV-15779 - mariabackup incremental prepare fails on CIFS mount.
CIFS does not like O_DIRECT flag (it is set successfully, but pread would
fail).

The fix is not to use O_DIRECT, there is not need for it.
posix_fadvise() was used already that should prevent buffer cache
pollution on Linux.

As recommended by documentation of posix_fadvise(), we'll also fsync()
tablespaces after a batch of writes.
2018-04-12 12:09:32 +01:00
Vladislav Vaintroub
15071226a0 MDEV-15780 : Mariabackup with absolute paths in innodb_data_file_path
System tablespace can be specified with absolute path, if innodb_data_home_dir
is empty.

This was not handled well  by mariabackup

1. In backup phase, empty innodb_data_home_dir variable from server was
not recognized. full paths were stored in backup-my.ini, even if
it stored all files locally. thus prepare phase would not find the system
tablespace files.

2. In copy-back phase, copy would not be done to the absolute destination
path, as path would be stripped with trim_dotslash

This patch fixes the above defects.
2018-04-12 11:15:27 +01:00
Vicențiu Ciorbaru
65eefcdc60 Merge remote-tracking branch '10.2' into 10.3 2018-04-12 12:41:19 +03:00
Vladislav Vaintroub
4c7a1a1b9e MDEV-15780 : mariabackup does not handle absolute names in for system tablespaces
Fix 10.2-specific bug - copy-back is not prepared to handle system
tablespaces with absolute path.
2018-04-11 23:25:45 +01:00
Vicențiu Ciorbaru
45e6d0aebf Merge branch '10.1' into 10.2 2018-04-10 17:43:18 +03:00