Commit graph

2245 commits

Author SHA1 Message Date
Sergei Golubchik
44f6f44593 Merge branch '10.0' into 10.1 2018-10-30 15:10:01 +01:00
Daniel Black
3859273d04 MDEV-14267: correct FSF address 2018-10-30 19:45:09 +08:00
Sergei Golubchik
57e0da50bb Merge branch '10.2' into 10.3 2018-09-28 16:37:06 +02:00
Marko Mäkelä
304857764b MDEV-13564 Mariabackup does not work with TRUNCATE
Implement undo tablespace truncation via normal redo logging.

Implement TRUNCATE TABLE as a combination of RENAME to #sql-ib name,
CREATE, and DROP.
2018-09-25 17:29:43 +03:00
Sergei Golubchik
5ae8fce50b Merge branch '10.1' into 10.2 2018-09-24 11:46:08 +02:00
Marko Mäkelä
8b6b2c3ea1 Fix mariabackup leaks (except my_load_defaults) 2018-09-21 11:18:59 +03:00
Vladislav Vaintroub
c139dc6d38 Windows : Fix application verifier errors.
Make different threads that are running data_copy_thread_func()
use the same pthread_mutex for locking (not the copies of the same
pthread_mutex_t).
2018-09-20 19:27:59 +01:00
Vladislav Vaintroub
0fa35ddf1f Amend fix for MDEV-17236
Simplify, and make it work with system tablespace outside of
innodb data home.

Also, do not reread TRX_SYS page  in endless loop,
if it appears to be corrupted.

Use finite number of attempts.
2018-09-20 14:08:57 +01:00
Thirunarayanan Balathandayuthapani
7e4bbd3aa6 MDEV-17236 mariabackup incorrectly tries to open ibdata1
if custom undo tablespace is defined

- In case of multiple undo tablespace, mariabackup have to open system
tablespace to find the list of undo tablespace present in TRX_SYS page.
For opening system tablespace, mariabackup should fetch the file name
from already initialized system tablespace object.
2018-09-20 17:13:12 +05:30
Marko Mäkelä
65474d92f5 Follow-up to "Fixed wrong printf in mariabackup"
This amends commit 4dc20ff687.

Starting with MariaDB 10.2, InnoDB defines
typedef size_t ulint;

The standard format for size_t uses the z modifier, for example,
"%zu" as in the macro ULINTPF.

"%lu" is wrong for size_t, because sizeof(unsigned long) can be
something else than sizeof(size_t). On Windows, the former would
always be 4 bytes, while size_t would be 4 or 8 bytes.
2018-09-18 06:23:25 +03:00
Michael Widenius
4dc20ff687 Fixed wrong printf in mariabackup 2018-09-16 11:23:27 +03:00
Vladislav Vaintroub
6b2da93359 MDEV-17192 Backup with -no-lock should fail, if DDL is detected at the end of backup 2018-09-14 09:36:02 +01:00
Oleksandr Byelkin
28f08d3753 Merge branch '10.1' into 10.2 2018-09-14 08:47:22 +02:00
Vladislav Vaintroub
e46b2a3e94 MDEV-12956 provide default datadir for mariabackup --copy-back
On Unix, it is compiled-in datadir value.
On Windows, the directory is ..\data, relative to directory
mariabackup.exe

server uses the same logic to determine datadir.
2018-09-11 20:59:35 +01:00
Vladislav Vaintroub
c3124174c3 MDEV-17168 mariabackup reports "failed to open bitmap directory"
MariaDB does not support changed page tracking, since 10.2. Remove bitmap
initialization
2018-09-11 15:24:35 +01:00
Marko Mäkelä
5a1868b58d MDEV-13564 Mariabackup does not work with TRUNCATE
This is a merge from 10.2, but the 10.2 version of this will not
be pushed into 10.2 yet, because the 10.2 version would include
backports of MDEV-14717 and MDEV-14585, which would introduce
a crash recovery regression: Tables could be lost on
table-rebuilding DDL operations, such as ALTER TABLE,
OPTIMIZE TABLE or this new backup-friendly TRUNCATE TABLE.
The test innodb.truncate_crash occasionally loses the table due to
the following bug:

MDEV-17158 log_write_up_to() sometimes fails
2018-09-07 22:15:06 +03:00
Marko Mäkelä
980d1bf1a9 MDEV-14717: Prevent crash-downgrade to earlier MariaDB 10.2
A crash-downgrade of a RENAME (or TRUNCATE or table-rebuilding
ALTER TABLE or OPTIMIZE TABLE) operation to an earlier 10.2 version
would trigger a debug assertion failure during rollback,
in trx_roll_pop_top_rec_of_trx(). In a non-debug build, the
TRX_UNDO_RENAME_TABLE record would be misinterpreted as an
update_undo log record, and typically the file name would be
interpreted as DB_TRX_ID,DB_ROLL_PTR,PRIMARY KEY. If a matching
record would be found, row_undo_mod() would hit ut_error in
switch (node->rec_type). Typically, ut_a(table2 == NULL) would
fail when opening the table from SQL.

Because of this, we prevent a crash-downgrade to earlier MariaDB 10.2
versions by changing the InnoDB redo log format identifier to the
10.3 identifier, and by introducing a subformat identifier so that
10.2 can continue to refuse crash-downgrade from 10.3 or later.
After a clean shutdown, a downgrade to MariaDB 10.2.13 or later would
still be possible thanks to MDEV-14909. A downgrade to older 10.2
versions is only possible after removing the log files (not recommended).

LOG_HEADER_FORMAT_CURRENT: Change to 103 (originally the 10.3 format).

log_group_t: Add subformat. For 10.2, we will use subformat 1,
and will refuse crash recovery from any other subformat of the
10.3 format, that is, a genuine 10.3 redo log.

recv_find_max_checkpoint(): Allow startup after clean shutdown
from a future LOG_HEADER_FORMAT_10_4 (unencrypted only).
We cannot handle the encrypted 10.4 redo log block format,
which was introduced in MDEV-12041. Allow crash recovery from
the original 10.2 format as well as the new format.
In Mariabackup --backup, do not allow any startup from 10.3 or 10.4
redo logs.

recv_recovery_from_checkpoint_start(): Skip redo log apply for
clean 10.3 redo log, but not for the new 10.2 redo log
(10.3 format, subformat 1).

srv_prepare_to_delete_redo_log_files(): On format or subformat
mismatch, set srv_log_file_size = 0, so that we will display the
correct message.

innobase_start_or_create_for_mysql(): Check for format or subformat
mismatch.

xtrabackup_backup_func(): Remove debug assertions that were made
redundant by the code changes in recv_find_max_checkpoint().
2018-09-07 22:10:03 +03:00
Marko Mäkelä
055a3334ad MDEV-13564 Mariabackup does not work with TRUNCATE
Implement undo tablespace truncation via normal redo logging.

Implement TRUNCATE TABLE as a combination of RENAME to #sql-ib name,
CREATE, and DROP.

Note: Orphan #sql-ib*.ibd may be left behind if MariaDB Server 10.2
is killed before the DROP operation is committed. If MariaDB Server 10.2
is killed during TRUNCATE, it is also possible that the old table
was renamed to #sql-ib*.ibd but the data dictionary will refer to the
table using the original name.

In MariaDB Server 10.3, RENAME inside InnoDB is transactional,
and #sql-* tables will be dropped on startup. So, this new TRUNCATE
will be fully crash-safe in 10.3.

ha_mroonga::wrapper_truncate(): Pass table options to the underlying
storage engine, now that ha_innobase::truncate() will need them.

rpl_slave_state::truncate_state_table(): Before truncating
mysql.gtid_slave_pos, evict any cached table handles from
the table definition cache, so that there will be no stale
references to the old table after truncating.

== TRUNCATE TABLE ==

WL#6501 in MySQL 5.7 introduced separate log files for implementing
atomic and crash-safe TRUNCATE TABLE, instead of using the InnoDB
undo and redo log. Some convoluted logic was added to the InnoDB
crash recovery, and some extra synchronization (including a redo log
checkpoint) was introduced to make this work. This synchronization
has caused performance problems and race conditions, and the extra
log files cannot be copied or applied by external backup programs.

In order to support crash-upgrade from MariaDB 10.2, we will keep
the logic for parsing and applying the extra log files, but we will
no longer generate those files in TRUNCATE TABLE.

A prerequisite for crash-safe TRUNCATE is a crash-safe RENAME TABLE
(with full redo and undo logging and proper rollback). This will
be implemented in MDEV-14717.

ha_innobase::truncate(): Invoke RENAME, create(), delete_table().
Because RENAME cannot be fully rolled back before MariaDB 10.3
due to missing undo logging, add some explicit rename-back in
case the operation fails.

ha_innobase::delete(): Introduce a variant that takes sqlcom as
a parameter. In TRUNCATE TABLE, we do not want to touch any
FOREIGN KEY constraints.

ha_innobase::create(): Add the parameters file_per_table, trx.
In TRUNCATE, the new table must be created in the same transaction
that renames the old table.

create_table_info_t::create_table_info_t(): Add the parameters
file_per_table, trx.

row_drop_table_for_mysql(): Replace a bool parameter with sqlcom.

row_drop_table_after_create_fail(): New function, wrapping
row_drop_table_for_mysql().

dict_truncate_index_tree_in_mem(), fil_truncate_tablespace(),
fil_prepare_for_truncate(), fil_reinit_space_header_for_table(),
row_truncate_table_for_mysql(), TruncateLogger,
row_truncate_prepare(), row_truncate_rollback(),
row_truncate_complete(), row_truncate_fts(),
row_truncate_update_system_tables(),
row_truncate_foreign_key_checks(), row_truncate_sanity_checks():
Remove.

row_upd_check_references_constraints(): Remove a check for
TRUNCATE, now that the table is no longer truncated in place.

The new test innodb.truncate_foreign uses DEBUG_SYNC to cover some
race-condition like scenarios. The test innodb-innodb.truncate does
not use any synchronization.

We add a redo log subformat to indicate backup-friendly format.
MariaDB 10.4 will remove support for the old TRUNCATE logging,
so crash-upgrade from old 10.2 or 10.3 to 10.4 will involve
limitations.

== Undo tablespace truncation ==

MySQL 5.7 implements undo tablespace truncation. It is only
possible when innodb_undo_tablespaces is set to at least 2.
The logging is implemented similar to the WL#6501 TRUNCATE,
that is, using separate log files and a redo log checkpoint.

We can simply implement undo tablespace truncation within
a single mini-transaction that reinitializes the undo log
tablespace file. Unfortunately, due to the redo log format
of some operations, currently, the total redo log written by
undo tablespace truncation will be more than the combined size
of the truncated undo tablespace. It should be acceptable
to have a little more than 1 megabyte of log in a single
mini-transaction. This will be fixed in MDEV-17138 in
MariaDB Server 10.4.

recv_sys_t: Add truncated_undo_spaces[] to remember for which undo
tablespaces a MLOG_FILE_CREATE2 record was seen.

namespace undo: Remove some unnecessary declarations.

fil_space_t::is_being_truncated: Document that this flag now
only applies to undo tablespaces. Remove some references.

fil_space_t::is_stopping(): Do not refer to is_being_truncated.
This check is for tablespaces of tables. Potentially used
tablespaces are never truncated any more.

buf_dblwr_process(): Suppress the out-of-bounds warning
for undo tablespaces.

fil_truncate_log(): Write a MLOG_FILE_CREATE2 with a nonzero
page number (new size of the tablespace in pages) to inform
crash recovery that the undo tablespace size has been reduced.

fil_op_write_log(): Relax assertions, so that MLOG_FILE_CREATE2
can be written for undo tablespaces (without .ibd file suffix)
for a nonzero page number.

os_file_truncate(): Add the parameter allow_shrink=false
so that undo tablespaces can actually be shrunk using this function.

fil_name_parse(): For undo tablespace truncation,
buffer MLOG_FILE_CREATE2 in truncated_undo_spaces[].

recv_read_in_area(): Avoid reading pages for which no redo log
records remain buffered, after recv_addr_trim() removed them.

trx_rseg_header_create(): Add a FIXME comment that we could write
much less redo log.

trx_undo_truncate_tablespace(): Reinitialize the undo tablespace
in a single mini-transaction, which will be flushed to the redo log
before the file size is trimmed.

recv_addr_trim(): Discard any redo logs for pages that were
logged after the new end of a file, before the truncation LSN.
If the rec_list becomes empty, reduce n_addrs. After removing
any affected records, actually truncate the file.

recv_apply_hashed_log_recs(): Invoke recv_addr_trim() right before
applying any log records. The undo tablespace files must be open
at this point.

buf_flush_or_remove_pages(), buf_flush_dirty_pages(),
buf_LRU_flush_or_remove_pages(): Add a parameter for specifying
the number of the first page to flush or remove (default 0).

trx_purge_initiate_truncate(): Remove the log checkpoints, the
extra logging, and some unnecessary crash points. Merge the code
from trx_undo_truncate_tablespace(). First, flush all to-be-discarded
pages (beyond the new end of the file), then trim the space->size
to make the page allocation deterministic. At the only remaining
crash injection point, flush the redo log, so that the recovery
can be tested.
2018-09-07 22:10:02 +03:00
Marko Mäkelä
4901f31c13 Merge 10.2 into 10.3 2018-09-07 22:09:28 +03:00
Vladislav Vaintroub
58389c71c2 MDEV-16671 - crash in mariabackup with my.cnf with plugin-load=ha_rocksdb
Remove plugin-load option from mariabackup. It does not needed to be an
option (we only need to store the plugin-load value during backup phase,
and reuse the same value during --prepare).

Fix is to read plugin-load from backup-my.cnf during prepare.
2018-09-07 18:18:49 +01:00
Marko Mäkelä
2f4c391958 Merge 10.2 into 10.3 2018-09-06 22:35:45 +03:00
Vladislav Vaintroub
a0631e7221 MDEV-17149 mariabackup hangs if innodb is not started
Fix exit condition for the log copying thread.
2018-09-06 15:31:29 +01:00
Sergei Golubchik
8bee7c16c8 compiler warnings (clang 4.0.1 on i386)
extra/mariabackup/fil_cur.cc:361:42: warning: format specifies type 'unsigned long' but the argument has type 'ib_int64_t' (aka 'long long') [-Wformat]
extra/mariabackup/fil_cur.cc:376:9: warning: format specifies type 'unsigned long' but the argument has type 'ib_int64_t' (aka 'long long') [-Wformat]
sql/handler.cc:6196:45: warning: format specifies type 'unsigned long' but the argument has type 'wsrep_trx_id_t' (aka 'unsigned long long') [-Wformat]
sql/log.cc:1681:16: warning: format specifies type 'unsigned long' but the argument has type 'size_t' (aka 'unsigned int') [-Wformat]
sql/log.cc:1687:16: warning: format specifies type 'unsigned long' but the argument has type 'size_t' (aka 'unsigned int') [-Wformat]
sql/wsrep_sst.cc:1388:86: warning: format specifies type 'long' but the argument has type 'wsrep_seqno_t' (aka 'long long') [-Wformat]
sql/wsrep_sst.cc:232:86: warning: format specifies type 'long' but the argument has type 'wsrep_seqno_t' (aka 'long long') [-Wformat]
storage/connect/filamdbf.cpp:450:47: warning: format specifies type 'short' but the argument has type 'int' [-Wformat]
storage/connect/filamdbf.cpp:970:47: warning: format specifies type 'short' but the argument has type 'int' [-Wformat]
storage/connect/inihandl.cpp:197:16: warning: address of array 'key->name' will always evaluate to 'true' [-Wpointer-bool-conversion]
storage/innobase/btr/btr0scrub.cc:151:17: warning: format specifies type 'long' but the argument has type 'int' [-Wformat]
storage/innobase/buf/buf0buf.cc:5085:8: warning: nonnull parameter 'bpage' will evaluate to 'true' on first encounter [-Wpointer-bool-conversion]
storage/innobase/fil/fil0crypt.cc:2454:5: warning: format specifies type 'long' but the argument has type 'int' [-Wformat]
storage/innobase/handler/ha_innodb.cc:18685:7: warning: format specifies type 'unsigned long' but the argument has type 'wsrep_trx_id_t' (aka 'unsigned long long') [-Wformat]
storage/innobase/row/row0mysql.cc:3319:5: warning: format specifies type 'long' but the argument has type 'int' [-Wformat]
storage/innobase/row/row0mysql.cc:3327:5: warning: format specifies type 'long' but the argument has type 'int' [-Wformat]
storage/maria/ma_norec.c:35:10: warning: implicit conversion from 'int' to 'my_bool' (aka 'char') changes value from 131 to -125 [-Wconstant-conversion]
storage/maria/ma_norec.c:42:10: warning: implicit conversion from 'int' to 'my_bool' (aka 'char') changes value from 131 to -125 [-Wconstant-conversion]
storage/maria/ma_test2.c:1009:12: warning: format specifies type 'unsigned long' but the argument has type 'size_t' (aka 'unsigned int') [-Wformat]
storage/maria/ma_test2.c:1010:12: warning: format specifies type 'unsigned long' but the argument has type 'size_t' (aka 'unsigned int') [-Wformat]
storage/mroonga/ha_mroonga.cpp:9189:44: warning: use of logical '&&' with constant operand [-Wconstant-logical-operand]
storage/mroonga/vendor/groonga/lib/expr.c:4987:22: warning: comparison of constant -1 with expression of type 'grn_operator' is always false [-Wtautological-constant-out-of-range-compare]
storage/xtradb/btr/btr0scrub.cc:151:17: warning: format specifies type 'long' but the argument has type 'int' [-Wformat]
storage/xtradb/buf/buf0buf.cc:5047:8: warning: nonnull parameter 'bpage' will evaluate to 'true' on first encounter [-Wpointer-bool-conversion]
storage/xtradb/fil/fil0crypt.cc:2454:5: warning: format specifies type 'long' but the argument has type 'int' [-Wformat]
storage/xtradb/row/row0mysql.cc:3324:5: warning: format specifies type 'long' but the argument has type 'int' [-Wformat]
storage/xtradb/row/row0mysql.cc:3332:5: warning: format specifies type 'long' but the argument has type 'int' [-Wformat]
unittest/sql/mf_iocache-t.cc:120:35: warning: format specifies type 'unsigned long' but the argument has type 'int' [-Wformat]
unittest/sql/mf_iocache-t.cc:96:35: note: expanded from macro 'INFO_TAIL'
2018-09-04 09:19:48 +02:00
Marko Mäkelä
7830fb7f45 Merge 10.2 into 10.3 2018-08-28 12:22:56 +03:00
Marko Mäkelä
d6f7fd6016 MDEV-13564: Refuse MLOG_TRUNCATE in mariabackup
The MySQL 5.7 TRUNCATE TABLE is inherently incompatible
with hot backup, because it is creating and deleting a separate
log file, and it is not writing redo log for all changes of the
InnoDB data dictionary tables. Refuse to create a corrupted backup
if the unsafe form of TRUNCATE was executed.

Note: Undo log tablespace truncation cannot be detected easily.
Also it is incompatible with backup, for similar reasons.

xtrabackup_backup_func(): "Subscribe to" the log events before
the first invocation of xtrabackup_copy_logfile().

recv_parse_or_apply_log_rec_body(): If the function pointer
log_truncate is set, invoke it to report MLOG_TRUNCATE.
2018-08-16 16:10:18 +03:00
Vladislav Vaintroub
f926c5f4fa MDEV-16996 mariabackup --prepare does not use native AIO on Linux by default 2018-08-16 08:37:54 +01:00
Marko Mäkelä
8716bb3b72 After-merge fix: Restore DECLARE_THREAD to fix Windows build 2018-08-16 09:41:20 +03:00
Marko Mäkelä
1eb2d8f6e8 Merge 10.2 into 10.3 2018-08-16 08:54:58 +03:00
Vladislav Vaintroub
922e7badfc MDEV-16791 mariabackup : Support DDL commands during backup 2018-08-14 15:11:13 +01:00
Vladislav Vaintroub
1faaaa9718 MDEV-15680 xb_aws_key_management fails in buildbot.
aws_key_management needs current directory to be datadir during
initalization, it scans current directory for encrypted keys.

Fix is to ensure, that plugin initialization in mariabackup happens
after the call to my_setwd(mysql_real_data_home).
2018-08-13 22:39:31 +01:00
Sergei Golubchik
0aa9b03393 Merge branch '10.2' into 10.3 2018-08-12 12:02:23 +02:00
Sergei Golubchik
675e7e7dcc remove obsolete checks for -fno-implicit-templates
we stopped using -fno-implicit-templates many years ago
2018-08-12 11:37:43 +02:00
Sergei Golubchik
ba1c05cc0d compiler warning
warning: suggest a space before ‘;’ or explicit braces around empty body in ‘for’ statement
2018-08-09 11:28:38 +02:00
Vladislav Vaintroub
3570ea9189 remove warning on Windows
Replace do-it-yourself version of strdup() with real strdup().
2018-08-05 00:36:59 +01:00
Marko Mäkelä
05459706f2 Merge 10.2 into 10.3 2018-08-03 15:57:23 +03:00
Marko Mäkelä
7b145fae13 mariabackup: Use snprintf() instead of sprintf() 2018-08-03 13:06:35 +03:00
Marko Mäkelä
814ae57daf Merge 10.1 into 10.2 2018-08-03 13:02:56 +03:00
Marko Mäkelä
769f6d2db7 Fix -Wclass-memaccess in WSREP,InnoDB,XtraDB 2018-08-03 12:21:13 +03:00
Marko Mäkelä
ef3070e997 Merge 10.1 into 10.2 2018-08-02 08:19:57 +03:00
Michael Widenius
0c745c743c Don't give warnings from perror or resolveip with safemalloc
- Added my_end(0) before exit
- Fixed typo in mysql_install_db
2018-07-29 10:56:11 +03:00
Sergei Golubchik
36e59752e7 Merge branch '10.2' into 10.3 2018-06-30 16:39:20 +02:00
Sergei Petrunia
f46acd4a3a Adopt Debian's fix-FTBFS-on-GNU-Hurd.patch.
- Took the original patch by Ondrej Sury;
- Applied a fix for a known problem in the patch:
   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=882062
- Fixed a few other issues
2018-06-29 14:00:00 +03:00
Vladislav Vaintroub
95ef8de891 mariabackup - rename backup-rocksdb option to rocksdb-backup
to avoid "using unique option prefix 'backup' is error-prone", there is
already --backup option there.
2018-06-22 23:30:26 +01:00
Vladislav Vaintroub
ecc4682672 MDEV-16519 : mariabackup should fail if MDL could not be acquired with lock-ddl-per-table
There is a tiny chance for race condition during MDL acquisition.

If table is renamed just prior to
SELECT 1 FROM <table_name> LIMIT 0

then this query would  fail, yet mariabackup --backup does not handle
it as fatal error and continues, only to fail later during file copy.

The fix is to die on error, of MDL lock query fails.
2018-06-22 15:24:09 +01:00
Marko Mäkelä
0121d5a790 Merge 10.2 into 10.3 2018-06-18 15:43:59 +03:00
Marko Mäkelä
b8514c94f6 MDEV-16496 Mariabackup: Implement --verbose option to instrument InnoDB log apply
srv_print_verbose_log: Introduce the value 2 to refer to
mariabackup --verbose.

recv_recover_page(), recv_parse_log_recs(): Add output for
mariabackup --verbose.
2018-06-15 16:14:12 +03:00
Marko Mäkelä
ff317fe08e Follow-up to MDEV-16367 mariabackup: error: failed to copy enough redo log
Commit dc9c555415 moved the final phase of
the redo log copying to the background thread. This would sometimes cause
too little redo log to be copied at the end of the backup. We would only
guarantee copying up to the latest redo log checkpoint. This would produce
a consistent backup, but it could refer to a too old point of time.

xtrabackup_copy_log(), xtrabackup_copy_logfile(): Add the parameter 'last'.

xtrabackup_backup_low(): Copy any remaining part of the log after the
backup threads have terminated.
2018-06-15 13:31:43 +03:00
Marko Mäkelä
a79b033b35 MDEV-16457 mariabackup 10.2+ should default to innodb_checksum_algorithm=crc32
Since MariaDB Server 10.2.2 (and MySQL 5.7), the default value of
innodb_checksum_algorithm is crc32 (CRC-32C), not the inefficient "innodb"
checksum. Change Mariabackup to use the same default, so that checksum
validation (when using the default algorithm on the server) will take less
time during mariabackup --backup. Also, mariabackup --prepare should be
a little faster, and the server should read backups faster, because the
page checksums would only be validated against CRC-32C.
2018-06-14 14:23:20 +03:00
Vladislav Vaintroub
aba2d7301f MDEV-13122 Backup myrocksdb with mariabackup. 2018-06-07 15:13:54 +01:00
Marko Mäkelä
62d21ddac1 Merge 10.2 into 10.3 2018-06-07 15:07:00 +03:00
Marko Mäkelä
dc9c555415 MDEV-16367 mariabackup: error: failed to copy enough redo log
log_copying_thread(): Keep copying redo log until the end has been
reached. (Previously, we would stop copying as soon as
the first batch of xtrabackup_copy_logfile() returned.)

log_copying: Remove. Use log_copying_running instead.

copy_logfile: Remove. Log copying will now only be invoked from
2 places: from xtrabackup_backup_func() for the initial batch,
and from log_copying_thread() until all of the log has been read.
Use the global variable metadata_to_lsn for determining if the
final part of the log is being copied.

xtrabackup_copy_log(): Add diagnostic messages for terminating
the copying. These messages should be dead code, because
log_group_read_log_seg() should be checking for the same.

xtrabackup_copy_logfile(): Correct the retrying logic.
If anything was successfully read, process the portion that
was read. On failure, let the caller close dst_log_file.

io_watching_thread(): Stop throttling during the last phase
of copying the log (metadata_to_lsn!=0). The final copying
of the log will now be performed in log_copying_thread().

stop_backup_threads(): Clean up the message about stopping
the log copying thread.

xtrabackup_backup_low(): Read metadata_to_lsn from the latest
checkpoint header page, even if it is the first page.
Let the log_copying_thread take care of copying all of
the redo log.
2018-06-07 14:29:35 +03:00
Marko Mäkelä
619c277a6c Mariabackup: Make some globals static 2018-06-07 14:11:55 +03:00
Yuqi
ee5124d714 Make MariaDB CRC32-lib platform independence (#780)
Make mariadb crc32 lib platform independent

It looks strange that someone can make use of 2 crc libraries
(Power64 or AArch64) at the same time.

The patch sets macros 'CRC32_LIBRARY' to make platform independence as an optional crc32 library.

Change-Id: I68bbf73cafb6a12f7fb105ad57d117b114a8c4af
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
2018-06-01 14:06:01 +04:00
Sergei Golubchik
4ec8598c1d Merge branch 'github/10.2' into 10.3 2018-05-22 11:47:09 +02:00
Sergei Golubchik
3a7d7e180a cleanup: create_temp_file()
simplify. move common code inside, specify common flags inside,
rewrite dead code (`if (mode & O_TEMPORARY)` on Linux, where
`O_TEMPORARY` is always 0) to actually do something.
2018-05-21 16:34:10 +00:00
Vladislav Vaintroub
207e5ba316 mariabackup : Fix race condition when killing query waiting for MDL
Itcan happen that the connection is already gone during the window
between quering I_S.PROCESSLIST and KILL QUERY.

Fix is to tolerate ER_NO_SUCH_THREAD returned from KILL QUERY.

Add small improvement in message "Killing MDL query " to actually output
the query.
Also do not try to kill queries that are already in Killed state.
2018-05-19 17:04:47 +00:00
Sergei Golubchik
c9717dc019 Merge branch '10.2' into 10.3 2018-05-11 13:15:10 +02:00
Sergei Golubchik
9b1824dcd2 Merge branch '10.1' into 10.2 2018-05-10 13:01:42 +02:00
Vladislav Vaintroub
f98496da96 MDEV-16105: Mariabackup does not support SSL
The reason is the missing HAVE_OPENSSL define for mariabackup.
2018-05-08 19:52:08 +00:00
Daniel Black
c28be510d1 Power8: use C implementation of crc32 instead of ASM
Compiles to same vector code, just a bit simplier.
vec_crc32.c is now identical to upstream
(https://github.com/antonblanchard/crc32-vpmsum/).

C code by Rogerio Alves <rogealve@br.ibm.com>

This implemention has been tested on big endian too.

Signed-off-by: Daniel Black <daniel@linux.ibm.com>
2018-05-07 09:26:12 +10:00
Marko Mäkelä
8bbcc0d505 MDEV-12218: Put back mariabackup --innodb-flush-method
Implement innodb_flush_method as an enum parameter in Mariabackup,
instead of ignoring the option and hard-wiring it to a default value.

xb0xb.h: Remove. Only xtrabackup.cc refers to the enum parameters.

innodb_flush_method_names[], innodb_flush_method_typelib[]:
Define as non-static, so that mariabackup can share the definitions.

srv_file_flush_method: Change the type to ulong, to match the
assignment in init_one_value() and handle_options() in mariabackup.
2018-04-30 18:22:52 +03:00
Marko Mäkelä
baa5a43d8c MDEV-16045: Replace log_group_t with log_t::files
There is only one log_sys and only one log_sys.log.

log_t::files::create(): Replaces log_init().

log_t::files::close(): Replaces log_group_close(), log_group_close_all().

fil_close_log_files(): if (free) log_sys.log_close();
The callers that passed free=true used to call log_group_close_all().

log_header_read(): Replaces log_group_header_read().

log_t::files::file_header_bufs_ptr: Use a single allocation.

log_t::files::file_header_bufs[]: Statically allocate the pointers.

log_t::files::set_fields(): Replaces log_group_set_fields().

log_t::files::calc_lsn_offset(): Replaces log_group_calc_lsn_offset().
Simplify the computation by using fewer variables.

log_t::files::read_log_seg(): Replaces log_group_read_log_seg().

log_sys_t::complete_checkpoint(): Replaces log_io_complete_checkpoint().

fil_aio_wait(): Move the logic from log_io_complete().
2018-04-29 09:46:24 +03:00
Marko Mäkelä
d73a898d64 MDEV-16045: Allocate log_sys statically
There is only one redo log subsystem in InnoDB. Allocate the object
statically, to avoid unnecessary dereferencing of the pointer.

log_t::create(): Renamed from log_sys_init().

log_t::close(): Renamed from log_shutdown().

log_t::checkpoint_buf_ptr: Remove. Allocate log_t::checkpoint_buf
statically.
2018-04-29 09:46:24 +03:00
Marko Mäkelä
715e4f4320 MDEV-12218 Clean up InnoDB parameter validation
Bind more InnoDB parameters directly to MYSQL_SYSVAR and
remove "shadow variables".

innodb_change_buffering: Declare as ENUM, not STRING.

innodb_flush_method: Declare as ENUM, not STRING.

innodb_log_buffer_size: Bind directly to srv_log_buffer_size,
without rounding it to a multiple of innodb_page_size.

LOG_BUFFER_SIZE: Remove.

SysTablespace::normalize_size(): Renamed from normalize().

innodb_init_params(): A new function to initialize and validate
InnoDB startup parameters.

innodb_init(): Renamed from innobase_init(). Invoke innodb_init_params()
before actually trying to start up InnoDB.

srv_start(bool): Renamed from innobase_start_or_create_for_mysql().
Added the input parameter create_new_db.

SRV_ALL_O_DIRECT_FSYNC: Define only for _WIN32.

xb_normalize_init_values(): Merge to innodb_init_param().
2018-04-29 09:41:42 +03:00
Marko Mäkelä
9ed2b2b2b8 Do not divide or multiply by srv_page_size
Instead, shift by srv_page_size_shift.
2018-04-28 20:52:22 +03:00
Marko Mäkelä
a90100d756 Replace univ_page_size and UNIV_PAGE_SIZE
Try to use one variable (srv_page_size) for innodb_page_size.

Also, replace UNIV_PAGE_SIZE_SHIFT with srv_page_size_shift.
2018-04-28 20:45:45 +03:00
Monty
2ccd6716fc Fix a lot of compiler warnings found by -Wunused 2018-04-26 17:35:12 +03:00
Marko Mäkelä
7396dfcca7 Merge 10.2 into 10.3 2018-04-24 20:59:57 +03:00
Marko Mäkelä
4cd7979c56 Merge 10.1 into 10.2 2018-04-24 09:39:45 +03:00
Marko Mäkelä
5b79303b40 MDEV-15988 Crash in ./mtr mariabackup.undo_space_id
xb_assign_undo_space_start(): Correctly pass the length of
the buffer, so that the file name will not be truncated.
2018-04-24 09:19:34 +03:00
Marko Mäkelä
6c64101bf0 MDEV-12266 follow-up fix to Mariabackup
xtrabackup_apply_delta(): Refer to fil_system.sys_space directly.
2018-04-23 13:14:28 +03:00
Marko Mäkelä
c6ba758d1d Merge 10.2 into 10.3 2018-04-23 09:49:58 +03:00
Marko Mäkelä
ea94717983 Merge 10.1 into 10.2 2018-04-21 11:58:32 +03:00
Michael Widenius
ddc5764303 Remove compiler warnings
- Remove unused variables
- Mark variables unused
- Fix wrong types
- Add no-strict-aliasing to BUILD scripts
2018-04-16 20:16:43 +03:00
Vladislav Vaintroub
c2dc72c0c3 MDEV-15779 - mariabackup incremental prepare fails on CIFS mount.
CIFS does not like O_DIRECT flag (it is set successfully, but pread would
fail).

The fix is not to use O_DIRECT, there is not need for it.
posix_fadvise() was used already that should prevent buffer cache
pollution on Linux.

As recommended by documentation of posix_fadvise(), we'll also fsync()
tablespaces after a batch of writes.
2018-04-12 12:09:32 +01:00
Vladislav Vaintroub
15071226a0 MDEV-15780 : Mariabackup with absolute paths in innodb_data_file_path
System tablespace can be specified with absolute path, if innodb_data_home_dir
is empty.

This was not handled well  by mariabackup

1. In backup phase, empty innodb_data_home_dir variable from server was
not recognized. full paths were stored in backup-my.ini, even if
it stored all files locally. thus prepare phase would not find the system
tablespace files.

2. In copy-back phase, copy would not be done to the absolute destination
path, as path would be stripped with trim_dotslash

This patch fixes the above defects.
2018-04-12 11:15:27 +01:00
Vicențiu Ciorbaru
65eefcdc60 Merge remote-tracking branch '10.2' into 10.3 2018-04-12 12:41:19 +03:00
Vladislav Vaintroub
4c7a1a1b9e MDEV-15780 : mariabackup does not handle absolute names in for system tablespaces
Fix 10.2-specific bug - copy-back is not prepared to handle system
tablespaces with absolute path.
2018-04-11 23:25:45 +01:00
Vicențiu Ciorbaru
45e6d0aebf Merge branch '10.1' into 10.2 2018-04-10 17:43:18 +03:00
Vladislav Vaintroub
ecf6675cfc MDEV-15713 mariabackup: throw warning, if --stream is used without --backup 2018-04-09 19:16:50 +01:00
Vladislav Vaintroub
37f24806fc MDEV-15825 Mariabackup help mentions Percona and PXC but not MariaDB 2018-04-09 16:22:15 +01:00
Sergey Vojtovich
5ccf3f96ac Fix misuse of MY_CHECK_CXX_COMPILER_FLAG
- compile_flags already include from top CMakeLists.txt
- MY_CHECK_CXX_COMPILER_FLAG() accepts only one parameter
- output variable of MY_CHECK_CXX_COMPILER_FLAG() is have_CXX__Wa__nH
- same check for mariabackup

Based on contribution by satanson (PR#466).
2018-04-05 17:45:36 +04:00
Marko Mäkelä
a5da1c64f8 Merge 10.2 into 10.3 2018-04-04 08:24:57 +03:00
Marko Mäkelä
bc2501453c Remove an unused variable 2018-04-03 16:58:35 +03:00
Thirunarayanan Balathandayuthapani
d9c5a46678 MDEV-15737 assertion in mariabackup.exe!recv_calc_lsn_on_data_add()
- recovered_lsn shouldn't be initialized during xtrabackup_copy_logfile().
If partial redo log read during the end of xtrabackup_copy_logfile() then
recovered_lsn will be different from scanned_lsn. Re-initialization of
recovered_lsn could lead to partial read again. It is a regression of
MDEV-14545
2018-04-03 16:43:36 +05:30
Vladislav Vaintroub
27c24808f7 MDEV-15636 mariabackup --lock-ddl-per-table hangs if ALTER table is running
concurrently.

There is a deadlock between

C1 mariabackup's connection that holds MDL locks
C2 Online ALTER TABLE that wants to have MDL exclusively
   and tries to upgrade its mdl lock.
C3 another mariabackup's connection that does FLUSH TABLES (or FTWRL)

C3 waits waits for C2,  which waits for C1, which waits for C3,
thus the deadlock.


MDL locks cannot be released until FLUSH  succeeds, because
otherwise it would allow ALTER to sneak in, causing backup to abort and
breaking lock-ddl-per-table's promise.

The fix here workarounds the deadlock, by killing connections in
"Waiting for metadata lock" status (i.e ALTER). This killing continues
until FTWRL succeeds.

Killing connections is skipped in case --no-locks parameter
was  passed to backup, because there won't be a FLUSH.

For the reference,in Percona's xtrabackup --lock-ddl-per-connection
silently implies --no-lock ie FLUSH is always skipped there.

A rather large part of fix is introducing DBUG capability to start
a query  the new connection at the right moment of backup
compensating somewhat for mariabackup' lack of send_query or DBUG_SYNC.
2018-04-01 14:26:06 +00:00
Marko Mäkelä
4cad42392a MDEV-12266: Change dict_table_t::space to fil_space_t*
InnoDB always keeps all tablespaces in the fil_system cache.
The fil_system.LRU is only for closing file handles; the
fil_space_t and fil_node_t for all data files will remain
in main memory. Between startup to shutdown, they can only be
created and removed by DDL statements. Therefore, we can
let dict_table_t::space point directly to the fil_space_t.

dict_table_t::space_id: A numeric tablespace ID for the corner cases
where we do not have a tablespace. The most prominent examples are
ALTER TABLE...DISCARD TABLESPACE or a missing or corrupted file.

There are a few functional differences; most notably:
(1) DROP TABLE will delete matching .ibd and .cfg files,
even if they were not attached to the data dictionary.
(2) Some error messages will report file names instead of numeric IDs.

There still are many functions that use numeric tablespace IDs instead
of fil_space_t*, and many functions could be converted to fil_space_t
member functions. Also, Tablespace and Datafile should be merged with
fil_space_t and fil_node_t. page_id_t and buf_page_get_gen() could use
fil_space_t& instead of a numeric ID, and after moving to a single
buffer pool (MDEV-15058), buf_pool_t::page_hash could be moved to
fil_space_t::page_hash.

FilSpace: Remove. Only few calls to fil_space_acquire() will remain,
and gradually they should be removed.

mtr_t::set_named_space_id(ulint): Renamed from set_named_space(),
to prevent accidental calls to this slower function. Very few
callers remain.

fseg_create(), fsp_reserve_free_extents(): Take fil_space_t*
as a parameter instead of a space_id.

fil_space_t::rename(): Wrapper for fil_rename_tablespace_check(),
fil_name_write_rename(), fil_rename_tablespace(). Mariabackup
passes the parameter log=false; InnoDB passes log=true.

dict_mem_table_create(): Take fil_space_t* instead of space_id
as parameter.

dict_process_sys_tables_rec_and_mtr_commit(): Replace the parameter
'status' with 'bool cached'.

dict_get_and_save_data_dir_path(): Avoid copying the fil_node_t::name.

fil_ibd_open(): Return the tablespace.

fil_space_t::set_imported(): Replaces fil_space_set_imported().

truncate_t: Change many member function parameters to fil_space_t*,
and remove page_size parameters.

row_truncate_prepare(): Merge to its only caller.

row_drop_table_from_cache(): Assert that the table is persistent.

dict_create_sys_indexes_tuple(): Write SYS_INDEXES.SPACE=FIL_NULL
if the tablespace has been discarded.

row_import_update_discarded_flag(): Remove a constant parameter.
2018-03-29 22:02:05 +03:00
Marko Mäkelä
05863142ad MDEV-12266: Remove fil_system_t::named_spaces
fil_space_get_by_name(): Remove.
(Implement differently in mariabackup.)

fil_ibd_open(): Check if the tablespace by the same ID already
exists. If it is the same name, return success, else failure.
2018-03-29 20:47:42 +03:00
Marko Mäkelä
2ac8b1a907 MDEV-12266: Add fil_system.sys_space, temp_space
Add fil_system_t::sys_space, fil_system_t::temp_space.
These will replace lookups for TRX_SYS_SPACE or SRV_TMP_SPACE_ID.

mtr_t::m_undo_space, mtr_t::m_sys_space: Remove.

mtr_t::set_sys_modified(): Remove.

fil_space_get_type(), fil_space_get_n_reserved_extents(): Remove.

fsp_header_get_tablespace_size(), fsp_header_inc_size():
Merge to the only caller, innobase_start_or_create_for_mysql().
2018-03-29 19:18:11 +03:00
Marko Mäkelä
600c85e85a Allocate the singleton fil_system statically
fil_system_t::create(): Replaces fil_init(), fsp_init().

fil_system_t::close(): Replaces fil_close().

fil_system_t::max_n_open: Remove. Use srv_max_n_open_files directly.
2018-03-29 19:18:10 +03:00
Sergei Golubchik
b1818dccf7 Merge branch '10.2' into 10.3 2018-03-28 17:31:57 +02:00
Sergei Golubchik
c764bc0a78 Merge branch '10.1' into 10.2 2018-03-25 13:02:52 +02:00
Sergei Golubchik
5fdbc3f66b compiler warning
extra/mariabackup/ds_buffer.c:145:9: warning: pointer targets in assignment differ in signedness [-Wpointer-sign]
2018-03-24 14:17:31 +01:00
Vladislav Vaintroub
af86422f08 MDEV-13023 mariabackup does not preserve holes for page compressed tables.
Changed "local" datasink logic to detect page compressed Innodb tables.

Whenever such table is detected, holes in the copied files are created by
skipping over binary zeros at the end of each compressed page.
2018-03-23 15:30:01 +00:00
Marko Mäkelä
e80a842000 Merge 10.1 into 10.2 2018-03-22 18:02:40 +02:00
Thirunarayanan Balathandayuthapani
b6d68c6aa3 MDEV-13561 Mariabackup is incompatible with retroactively created innodb_undo_tablespaces
- Mariabackup supports starting undo tablespace id which is greater
than 1.
2018-03-22 14:19:16 +05:30
Marko Mäkelä
865cec928a MDEV-15505 Fix wsrep XID storage byte order 2018-03-21 16:29:30 +02:00
Teemu Ollakka
33aad1d273 MDEV-15505 Fixes to compilation without -DWITH_WSREP:BOOL=ON
Removed including wsrep_api.h from service_wsrep.h. This caused
various kinds of collisions with definitions when wsrep is
not supposed to be built in. Defined functions wsrep_xid_seqno()
and wsrep_xid_uuid() in wsrep_dummy.cc. Replaced wsrep_seqno_t
with long long where wsrep_api.h is not included.

Removed wsrep_xid_seqno() macro from wsrep_mysqld.h and made
wsrep code using wsrep_xid_seqno() in handler.cc to be compiled
in only if WITH_WSREP is ON.

Included wsrep_api.h for mariabackup if WITH_WSREP is ON.
2018-03-21 12:02:09 +02:00
Marko Mäkelä
613be24b7a Merge 10.0 into 10.1 2018-03-20 19:25:08 +02:00
Marko Mäkelä
0492100059 Merge 5.5 into 10.0 2018-03-20 18:36:03 +02:00
Thirunarayanan Balathandayuthapani
6d1d5c3aeb MDEV-14545 Backup fails due to MLOG_INDEX_LOAD record
- Fixed the asan failure of the unsupported_redo test case
2018-03-16 20:55:55 +05:30
Marko Mäkelä
bd7ed1b923 MDEV-13935 INSERT stuck at state Unlocking tables
Revert the dead code for MySQL 5.7 multi-master replication (GCS),
also known as
WL#6835: InnoDB: GCS Replication: Deterministic Deadlock Handling
(High Prio Transactions in InnoDB).

Also, make innodb_lock_schedule_algorithm=vats skip SPATIAL INDEX,
because the code does not seem to be compatible with them.

Add FIXME comments to some SPATIAL INDEX locking code. It looks
like Galera write-set replication might not work with SPATIAL INDEX.
2018-03-16 15:50:04 +02:00
Marko Mäkelä
84129fb1b5 After-merge fix for commit 98eb9518db
The merge only covered 10.1 up to
commit 4d248974e0.

Actually merge the changes up to
commit 0a534348c7.

Also, remove the unused InnoDB field trx_t::abort_type.
2018-03-16 15:49:53 +02:00
Sergey Vojtovich
0a534348c7 MDEV-14265 - RPMLint warning: shared-lib-calls-exit
Eliminated last exit() call from libmysqld.
2018-03-16 13:26:52 +04:00
Vladislav Vaintroub
0943b33de3 MDEV-12190 YASSL isn't able to negotiate TLS version correctly
Backport from 10.2
2018-03-14 14:35:27 +00:00
Thirunarayanan Balathandayuthapani
ff909acfa4 MDEV-14545 Backup fails due to MLOG_INDEX_LOAD record
Problem:
=======
  Mariabackup exits during prepare phase if it encounters
MLOG_INDEX_LOAD redo log record. MLOG_INDEX_LOAD record
informs Mariabackup that the backup cannot be completed based
on the redo log scan, because some information is purposely
omitted due to bulk index creation in ALTER TABLE.

Solution:
========
Detect the MLOG_INDEX_LOAD redo record during backup phase and
exit the mariabackup with the proper error message.
2018-03-13 15:25:38 +05:30
Marko Mäkelä
7fb03d7abf Merge bb-10.2-ext into 10.3 2018-03-13 08:15:06 +02:00
Marko Mäkelä
e0792e7603 Merge 10.2 into bb-10.2-ext 2018-03-12 15:11:15 +02:00
Teemu Ollakka
b125ae0a84 MDEV-15505 New wsrep XID format for backwards compatibility
A new wsrep XID format was added to keep the XID implementation
backwards compatible. Original version always reads XID seqno
part in host byte order, the new version in little endian byte
order. Wsrep XID will always be written in the new format.

Included wsrep_api.h from service_wsrep.h for wsrep type definitions.
Removed redundant wsrep XID code from mariabackup and included
service_wsrep.h in order to use
2018-03-12 14:51:49 +02:00
Teemu Ollakka
dd74b94823 MDEV-15505 Fix wsrep XID seqno byte order
The problem is that the seqno part of wsrep XID is always
stored in host byte order. This may cause issues when a physical
backup is restored on a host with different architecture, the
seqno part with XID may have incorrect value.

In order to fix this, wsrep XID seqno is always written into
XID data buffer in little endian byte order using int8store()
and read from data buffer using sint8korr(). For backwards
compatibility the seqno is read from TRX_SYS page in host
byte order during upgrade.

This patch implements byte ordering in wsrep_xid_init(),
wsrep_xid_seqno(), and exposes functions to read wsrep
XID uuid and seqno in wsrep_service_st. Backwards compatibility
for upgrade is provided in trx_rseg_init_wsrep_xid().
2018-03-12 14:46:20 +02:00
Marko Mäkelä
54765aaa4d MDEV-15524 Do not disable page checksums for temporary tables
buf_flush_init_for_writing(): Remove the parameter skip_checksum.
2018-03-10 11:54:34 +02:00
Sergey Vojtovich
131d9a5d0c Allocate lock_sys statically
There is only one lock_sys. Allocate it statically in order to avoid
dereferencing a pointer whenever accessing it. Also, align some
members to their own cache line in order to avoid false sharing.

lock_sys_t::create(): The deferred constructor.

lock_sys_t::close(): The early destructor.
2018-02-23 08:18:18 +02:00
Vladislav Vaintroub
56e7b7eaed Make possible to use clang on Windows (clang-cl)
-DWITH_ASAN can be used as well now, on x64

Fix many clang-cl warnings.
2018-02-20 21:17:36 +00:00
Marko Mäkelä
947efe17ed MDEV-15158 On commit, do not write to the TRX_SYS page
This is based on a prototype by
Thirunarayanan Balathandayuthapani <thiru@mariadb.com>.

Binlog and Galera write-set replication information was written into
TRX_SYS page on each commit. Instead of writing to the TRX_SYS during
normal operation, InnoDB can make use of rollback segment header pages,
which are already being written to during a commit.

The following list of fields in rollback segment header page are added:
    TRX_RSEG_BINLOG_OFFSET
    TRX_RSEG_BINLOG_NAME (NUL-terminated; empty name = not present)
    TRX_RSEG_WSREP_XID_FORMAT (0=not present; 1=present)
    TRX_RSEG_WSREP_XID_GTRID
    TRX_RSEG_WSREP_XID_BQUAL
    TRX_RSEG_WSREP_XID_DATA

trx_sys_t: Introduce the fields
recovered_binlog_filename, recovered_binlog_offset, recovered_wsrep_xid.

To facilitate upgrade from older mysql or mariaDB versions, we will read
the information in TRX_SYS page. It will be overridden by the
information that we find in rollback segment header pages.

Mariabackup --prepare will read the metadata from the rollback
segment header pages via trx_rseg_array_init(). It will still
not read any undo log pages or recover any transactions.
2018-02-20 21:36:36 +02:00
Marko Mäkelä
2ba487cfe8 Merge bb-10.2-ext into 10.3 2018-02-19 11:37:29 +02:00
Marko Mäkelä
278c036275 Merge 10.2 into bb-10.2-ext 2018-02-19 09:01:06 +02:00
Vladislav Vaintroub
2129eab7e2 MDEV-15071 backup does not store xtrabackup_info in the --extra-lsndir directory 2018-02-15 21:08:18 +00:00
Vladislav Vaintroub
a08121c978 MDEV-14997 mariabackup crashes with invalid --innodb-flush-method
Avoid NULL pointer dereference when shutting down after invalid parameters
were parsed.
2018-02-15 17:40:14 +00:00
Marko Mäkelä
b006d2ead4 Merge bb-10.2-ext into 10.3 2018-02-15 10:22:03 +02:00
Marko Mäkelä
0c4aeef976 Merge 10.2 into bb-10.2-ext 2018-02-13 16:51:45 +02:00
Marko Mäkelä
00f0c039d2 MDEV-15270 Mariabackup should not try to use doublewrite buffer
When Mariabackup gets a bad read of the first page of the system
tablespace file, it would inappropriately try to apply the doublewrite
buffer and write changes back to the data file (to the source file)!
This is very wrong and must be prevented.

The correct action would be to retry reading the system tablespace
as well as any other files whose first page was read incorrectly.
Fixing this was not attempted.

xb_load_tablespaces(): Shorten a bogus message to be more relevant.
The message can be displayed by --backup or --prepare.

xtrabackup_backup_func(), os_file_write_func(): Add a missing space
to a message.

Datafile::restore_from_doublewrite(): Do not even attempt the
operation in Mariabackup.

recv_init_crash_recovery_spaces(): Do not attempt to restore the
doublewrite buffer in Mariabackup (--prepare or --export), because
all pages should have been copied correctly in --backup already,
and because --backup should ignore the doublewrite buffer.

SysTablespace::read_lsn_and_check_flags(): Do not attempt to initialize
the doublewrite buffer in Mariabackup.

innodb_make_page_dirty(): Correct the bounds check.

Datafile::read_first_page(): Correct the name of the parameter.
2018-02-12 16:56:01 +02:00
Alexander Barkov
3cad31f2a7 Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext 2018-02-08 19:06:25 +04:00
Sergei Golubchik
4771ae4b22 Merge branch 'github/10.1' into 10.2 2018-02-06 14:50:50 +01:00
Vladislav Vaintroub
6c279ad6a7 MDEV-15091 : Windows, 64bit: reenable and fix warning C4267 (conversion from 'size_t' to 'type', possible loss of data)
Handle string length as size_t, consistently (almost always:))
Change function prototypes to accept size_t, where in the past
ulong or uint were used. change local/member variables to size_t
when appropriate.

This fix excludes rocksdb, spider,spider, sphinx and connect for now.
2018-02-06 12:55:58 +00:00
Marko Mäkelä
d7d910d08b Fix a warning about possibly unused variable
xtrabackup_init_temp_log(): Remove a redundant variable and limit scope.
2018-02-03 12:50:38 +02:00
Sergei Golubchik
d4df7bc9b1 Merge branch 'github/10.0' into 10.1 2018-02-02 10:09:44 +01:00
Daniel Black
7eb084fe2c MariaBackup: gcc7 - snprintf output overflow warning
extra/mariabackup/xtrabackup.cc: In function ‘ulint xb_process_datadir(const char*, const char*, handle_datadir_entry_func_t)’:
extra/mariabackup/xtrabackup.cc:4534:1: warning: ‘snprintf’ output may be truncated before the last format character [-Wformat-truncation=]
 xb_process_datadir(
 ^~~~~~~~~~~~~~~~~~
mariabackup/xtrabackup.cc:4607:11: note: ‘snprintf’ output 2 or more bytes (assuming 4001) into a destination of size 4000
   snprintf(dbpath, sizeof(dbpath), "%s/%s", path, dbinfo.name);
   ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2018-01-31 11:43:59 +01:00
Marko Mäkelä
8d1d38f953 Simplify access to the TRX_SYS page
trx_sysf_t: Remove.

trx_sysf_get(): Return the TRX_SYS page, not a pointer within it.

trx_sysf_rseg_get_space(), trx_sysf_rseg_get_page_no():
Remove a parameter, and merge the declaration and definition.
Take the TRX_SYS page as a parameter.

TRX_SYS_N_RSEGS: Correct the comment.

trx_sysf_rseg_find_free(), trx_sys_update_mysql_binlog_offset(),
trx_sys_update_wsrep_checkpoint(): Take the TRX_SYS page as a parameter.

trx_rseg_header_create(): Add a parameter for the TRX_SYS page.

trx_sysf_rseg_set_space(), trx_sysf_rseg_set_page_no(): Remove;
merge to the only caller, trx_rseg_header_create().
2018-01-31 08:52:16 +02:00
Monty
a7e352b54d Changed database, tablename and alias to be LEX_CSTRING
This was done in, among other things:
- thd->db and thd->db_length
- TABLE_LIST tablename, db, alias and schema_name
- Audit plugin database name
- lex->db
- All db and table names in Alter_table_ctx
- st_select_lex db

Other things:
- Changed a lot of functions to take const LEX_CSTRING* as argument
  for db, table_name and alias. See init_one_table() as an example.
- Changed some function arguments from LEX_CSTRING to const LEX_CSTRING
- Changed some lists from LEX_STRING to LEX_CSTRING
- threads_mysql.result changed because process list_db wasn't always
  correctly updated
- New append_identifier() function that takes LEX_CSTRING* as arguments
- Added new element tmp_buff to Alter_table_ctx to separate temp name
  handling from temporary space
- Ensure we store the length after my_casedn_str() of table/db names
- Removed not used version of rename_table_in_stat_tables()
- Changed Natural_join_column::table_name and db_name() to never return
  NULL (used for print)
- thd->get_db() now returns db as a printable string (thd->db.str or "")
2018-01-30 21:33:55 +02:00
Marko Mäkelä
33714d2065 Merge bb-10.2-ext into 10.3 2018-01-30 21:04:48 +02:00
Marko Mäkelä
6d390bab4a Merge 10.2 into bb-10.2-ext 2018-01-30 20:18:25 +02:00
Vladislav Vaintroub
7cdf759c86 MDEV-14485 Server hangs on startup in THD::init
Solve 3 way deadlock between plugin_initialiaze(), THD::init() and
mysql_sys_var_char().

The deadlock exists because of the lock order inversion between
LOCK_global_system_variables mutex and LOCK_system_variables_hash
read-write lock-

In this case, it is enough to change LOCK_system_variables_hash to prefer
reads to fix the deadlock, i.e change it to mysql_prlock_t
2018-01-29 07:33:52 +00:00
Marko Mäkelä
9875d5c3e1 Merge bb-10.2-ext into 10.3 2018-01-24 14:00:33 +02:00
Marko Mäkelä
62740e02c8 Merge 10.2 into bb-10.2-ext 2018-01-24 11:15:11 +02:00
Daniel Black
12f900228f mariadbbackup: use defaults-group-suffix even if no --defaults-file
Signed-off-by: Daniel Black <daniel@linux.vnet.ibm.com>
2018-01-23 22:57:54 +01:00
Alexander Barkov
ec6b8c546a Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext 2018-01-23 17:43:12 +04:00
Vladislav Vaintroub
87db5eb813 MDEV-13825 mariabackup --lock-ddl-per-table does not properly lock
FULLTEXT auxiliary tables

Change the logic to take mdl lock on all tables before tablespaces are
copied, rather than  lock every single tablespace just before it is
copied.
2018-01-23 09:13:12 +00:00
Vladislav Vaintroub
30b1fbda28 MDEV-14746 : read [mariabackup] section from config file. 2018-01-22 20:16:19 +00:00
Vladislav Vaintroub
054051bb76 MDEV-14150 - backup should fail early if rsync is missing
check availability of rsync utility early , when starting backup with
--rsync. Fail if it is not there.
2018-01-22 20:01:03 +00:00
Vladislav Vaintroub
e30cdb68e5 mariabackup : improve help text 2018-01-22 15:10:56 +00:00
Marko Mäkelä
2281fcf38a Follow-up fix to MDEV-14952 for Mariabackup
innodb_init_param(): Initialize btr_ahi_parts=1 for Mariabackup.

btr_search_enabled: Let the adaptive hash index be disabled
in Mariabackup. This would potentially only matter during --export,
and --export performs a table scan, not many index lookups.
2018-01-16 14:08:48 +02:00
Marko Mäkelä
16d308e21d MDEV-14874 innodb_encrypt_log corrupts the log when the LSN crosses 32-bit boundary
This bug affects both writing and reading encrypted redo log in
MariaDB 10.1, starting from version 10.1.3 which added support for
innodb_encrypt_log. That is, InnoDB crash recovery and Mariabackup
will sometimes fail when innodb_encrypt_log is used.

MariaDB 10.2 or Mariabackup 10.2 or later versions are not affected.

log_block_get_start_lsn(): Remove. This function would cause trouble if
a log segment that is being read is crossing a 32-bit boundary of the LSN,
because this function does not allow the most significant 32 bits of the
LSN to change.

log_blocks_crypt(), log_encrypt_before_write(), log_decrypt_after_read():
Add the parameter "lsn" for the start LSN of the block.

log_blocks_encrypt(): Remove (unused function).
2018-01-08 09:44:40 +02:00
Monty
5e0b13d173 Fixed wrong arguments to printf and related functions
Other things, mainly to get
create_mysqld_error_find_printf_error tool to work:

- Added protection to not include mysqld_error.h twice
- Include "unireg.h" instead of "mysqld_error.h" in server
- Added protection if ER_XX messages are already defined
- Removed wrong calls to my_error(ER_OUTOFMEMORY) as
  my_malloc() and my_alloc will do this automatically
- Added missing %s to ER_DUP_QUERY_NAME
- Removed old and wrong calls to my_strerror() when using
  MY_ERROR_ON_RENAME (wrong merge)
- Fixed deadlock error message from Galera. Before the extra
  information given to ER_LOCK_DEADLOCK was missing because
  ER_LOCK_DEADLOCK doesn't provide any extra information.

I kept #ifdef mysqld_error_find_printf_error_used in sql_acl.h
to make it easy to do this kind of check again in the future
2018-01-04 16:24:09 +02:00
Marko Mäkelä
145ae15a33 Merge bb-10.2-ext into 10.3 2018-01-04 09:22:59 +02:00
Monty
c2c2173727 Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext 2018-01-03 02:59:32 +02:00
Monty
83a8b0e9d3 Fixed compiler warning 2018-01-03 01:01:53 +02:00
Monty
fbab79c9b8 Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext
Conflicts:
	cmake/make_dist.cmake.in
	mysql-test/r/func_json.result
	mysql-test/r/ps.result
	mysql-test/t/func_json.test
	mysql-test/t/ps.test
	sql/item_cmpfunc.h
2018-01-01 19:39:59 +02:00
Vicențiu Ciorbaru
985d2d393c Merge remote-tracking branch 'origin/10.1' into 10.2 2017-12-22 12:23:39 +02:00
Marko Mäkelä
0436a0ff3c Merge bb-10.2-ext into 10.3 2017-12-19 17:28:22 +02:00
Marko Mäkelä
028e91f380 Merge 10.2 into bb-10.2-ext 2017-12-19 17:12:14 +02:00
Sergey Vojtovich
2cd3169113 MDEV-14265 - RPMLint warning: shared-lib-calls-exit
find_type_or_exit() client helper did exit(1) on error, exit(1) moved to
clients.

mysql_read_default_options() did exit(1) on error, error is passed through and
handled now.

my_str_malloc_default() did exit(1) on error, replaced my_str_ allocator
functions with normal my_malloc()/my_realloc()/my_free().

sql_connect.cc did many exit(1) on hash initialisation failure. Removed error
check since my_hash_init() never fails.

my_malloc() did exit(1) on error. Replaced with abort().

my_load_defaults() did exit(1) on error, replaced with return 2.

my_load_defaults() still does exit(0) when invoked with --print-defaults.
2017-12-19 19:10:54 +04:00
Marko Mäkelä
8d70097c21 Merge 10.1 to 10.2
Follow-up fix to MDEV-14008: Let Field_double::val_uint() silently
return 0 on error
2017-12-19 16:48:28 +02:00
Vladislav Vaintroub
f32063c513 MDEV-13620 - improve help message for 'plugin-dir' and 'plugin-load'
options.
2017-12-18 15:37:06 +00:00
Marko Mäkelä
a9a4089175 Plug a small memory leak in mariabackup --backup 2017-12-14 13:47:38 +02:00
Vesa Pentti
99bcec295d MDEV-12501 -- set --maturity-level by default
* Note: breaking change; since this commit, a plugin that has
    worked so far might get rejected due to plugin maturity
  * mariabackup is not affected (allows all plugins)
  * VERSION file defines SERVER_MATURITY, which defines the
    corresponding numeric value as SERVER_MATURITY_LEVEL in
    include/mysql_version.h
  * The default value for 'plugin_maturity' is SERVER_MATURITY_LEVEL - 1
  * Logs a warning if a plugin has maturity lower than
    SERVER_MATURITY_LEVEL
  * Tests suppress the plugin maturity warning
  * Tests use --plugin-maturity=unknown by default so as not to fail
    due to the stricter plugin maturity handling
2017-12-09 23:34:43 +00:00
Marko Mäkelä
976f6fb1b6 Merge bb-10.2-ext into 10.3 2017-12-06 19:36:33 +02:00
Marko Mäkelä
ce07676502 Merge 10.2 into bb-10.2-ext 2017-12-06 19:34:03 +02:00
Vesa Pentti
5868a184fa Revert "MDEV-12501 -- set --maturity-level by default"
This reverts commit 1af2d7ba23.
2017-12-05 08:49:28 +00:00
Vesa Pentti
1af2d7ba23 MDEV-12501 -- set --maturity-level by default
* Note: breaking change; since this commit, a plugin that has
    worked so far might get rejected due to plugin maturity
  * mariabackup is not affected (allows all plugins)
  * VERSION file defines SERVER_MATURITY, which defines the
    corresponding numeric value as SERVER_MATURITY_LEVEL in
    include/mysql_version.h
  * The default value for 'plugin_maturity' is SERVER_MATURITY_LEVEL - 1
  * Logs a warning if a plugin has maturity lower than
    SERVER_MATURITY_LEVEL
  * Tests suppress the plugin maturity warning
  * Tests use --plugin-maturity=unknown by default so as not to fail
    due to the stricter plugin maturity handling
2017-12-04 21:12:35 +00:00
Monty
60df17e95a Remove compiler warnings 2017-12-03 13:58:36 +02:00
Monty
d9188adae5 resolve_stack_dump updated to match latest stack trace format
Originally only symbols withing [] where resolved. Now we resolve
symbols also withing (+...)

To make it easier to see where the resolved symbol comes from, we
resolve the symbol 'in place' instead of just printing the resolved
symbol alone.
2017-12-03 12:45:54 +02:00
Marko Mäkelä
ddac2d7a1e Merge bb-10.2-ext into 10.3 2017-12-01 15:37:30 +02:00
Marko Mäkelä
24c9785a67 Merge 10.2 into bb-10.2-ext 2017-12-01 15:35:16 +02:00
Vladislav Vaintroub
f59a1826f8 MDEV-14536 : during backup, retry read of log blocks, if there is
(possibly intermittent) checksum mismatch.
2017-11-30 16:30:40 +00:00
Marko Mäkelä
7cb3520c06 Merge bb-10.2-ext into 10.3 2017-11-30 08:16:37 +02:00
Vladislav Vaintroub
bf6d11c4d6 MDEV-14536 : In mariabackup, reread redo log blocks , if checksum mismatch
is detected.

The checksum mismatch can be due to partial write, thus retry the read
2017-11-29 14:53:12 +00:00
Alexander Barkov
5b697c5a23 Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext 2017-11-29 12:06:48 +04:00
Marko Mäkelä
23d2dae5f0 Fix some integer type mismatch warnings 2017-11-28 18:29:20 +02:00
Marko Mäkelä
447cd7b1af Test MDL with a more generic ALTER TABLE statement 2017-11-27 11:20:04 +02:00
Vladislav Vaintroub
414d3a3e17 Fix warning. 2017-11-24 17:33:53 +00:00
Sergey Vojtovich
4cc20c88df Fixed build failure with PFS disabled
mariabackup fails to builds with PFS disabled. The reason was missing include,
which was included by PFS otherwise.
2017-11-24 17:30:40 +00:00
Marko Mäkelä
f1cc6e3874 Merge 10.1 into 10.2 2017-11-24 17:17:16 +02:00
Marko Mäkelä
6979d20426 MDEV-14499 Mariabackup 10.2 fails to back up a multi-file InnoDB system tablespace
When Mariabackup is invoked on an instance that uses a multi-file
InnoDB system tablespace, it may fail to other files of the system
tablespace than the first one.
This was revealed by the MDEV-14447 test case.
The offending code is assuming that the first page of each data file
is page 0. But, in multi-file system tablespaces that is not the case.

xb_fil_cur_open(): Instead of re-reading the first page of the file,
rely on the fil_space_t metadata that already exists in memory.

xb_get_space_flags(): Remove.
2017-11-24 17:12:38 +02:00
Sergey Vojtovich
abf61fd91f Fixed build failure with PFS disabled
mariabackup fails to builds with PFS disabled. The reason was missing include,
which was included by PFS otherwise.
2017-11-24 15:01:55 +04:00
Vladislav Vaintroub
316f0d8fe3 MDEV-14447 mariabackup incremental incorrectly extends system tablespace
for multi-file innodb_data_file_path.

Use fil_extend_space_to_desired_size() to correctly extend system
tablespace. Make sure to get tablespace size from the first tablespace
part.
2017-11-24 00:01:29 +00:00
Sergei Golubchik
7f1900705b Merge branch '10.1' into 10.2 2017-11-21 19:47:46 +01:00
Vladislav Vaintroub
7c4f859384 MDEV-14283 : Fix compilation of mariabackup for gcc3.x 2017-11-21 17:19:32 +00:00
Alexander Barkov
4a8039b04e Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext 2017-11-20 11:12:08 +04:00
Marko Mäkelä
c19ef508b8 InnoDB: Remove ut_snprintf() and the use of my_snprintf(); use snprintf() 2017-11-13 02:11:48 +02:00
Marko Mäkelä
a48aa0cd56 Merge bb-10.2-ext into 10.3 2017-11-10 16:12:45 +02:00
Monty
0bb0d52221 Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext
Conflicts:
	mysql-test/r/cte_recursive.result
	mysql-test/r/derived_cond_pushdown.result
	mysql-test/t/cte_recursive.test
	mysql-test/t/derived_cond_pushdown.test
	sql/datadict.cc
	sql/handler.cc
2017-11-09 23:21:41 +02:00
Vladislav Vaintroub
53c7aaf332 MDEV-14077 Incremental backup extremly slow
Remove the main bottleneck -  the memset() call mentioned in the bug.
Use os_mem_alloc_large() instead of malloc()/memset().
2017-11-09 19:07:05 +00:00
Marko Mäkelä
7c85a8d936 Merge 10.1 into 10.2 2017-11-08 13:12:11 +02:00
Marko Mäkelä
843e4508c0 Merge 10.1 into 10.2 2017-11-07 23:02:39 +02:00
Vladislav Vaintroub
ffb1eebe05 MDEV-13560 Copy all innodb undo tablespaces from the backup directory to destination.
Do not check or rely on innodb_undo_tablespaces in copy-back,
there is no good reason for that.
2017-11-03 16:02:19 +00:00
Alexander Barkov
835cbbcc7b Merge remote-tracking branch 'origin/bb-10.2-ext' into 10.3
TODO: enable MDEV-13049 optimization for 10.3
2017-10-30 20:47:39 +04:00
Alexander Barkov
003cb2f424 Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext 2017-10-30 16:42:46 +04:00
Marko Mäkelä
cbd0da66e4 Merge 10.1 into 10.2 2017-10-25 17:17:21 +03:00
Marko Mäkelä
909cdafd35 MDEV-13496 Use "mariabackup" rather than "xtrabackup" in console output
Replace the remaining "xtrabackup:" references in the code.
2017-10-25 09:06:45 +03:00
Sergei Golubchik
e0a1c745ec Merge branch '10.1' into 10.2 2017-10-24 14:53:18 +02:00
Vladislav Vaintroub
72407e544e MDEV-13496 Use "mariabackup" rather than "xtrabackup" in console output 2017-10-23 10:37:28 +00:00
Vladislav Vaintroub
125ce6f82f MDEV-14102 restore --remove-original options for mariabackup 2017-10-23 10:30:17 +00:00
Sergei Golubchik
2aa51f528f Various compier warnings
gcc 5.4 and 7.1, Debug and Release builds
2017-10-22 14:51:45 +02:00
Sergei Golubchik
9d2e2d7533 Merge branch '10.0' into 10.1 2017-10-22 13:03:41 +02:00
Sergei Golubchik
da4503e956 Merge branch '5.5' into 10.0 2017-10-18 15:14:39 +02:00
Alexander Barkov
30e7d6709f Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext 2017-10-18 14:11:55 +04:00
Sergei Golubchik
df5f25fa7a Merge branch 'mysql/5.5' into 5.5 2017-10-17 10:18:17 +02:00
Sergei Golubchik
d76f5774fe MDEV-13459 Warnings, when compiling with gcc-7.x
mostly caused by -Wimplicit-fallthrough
2017-10-17 07:37:39 +02:00
Marko Mäkelä
9ee840cd0a mariabackup: Properly call os_thread_exit() with detach=true
There is no call to os_thread_join(), so we should detach the thread
handles in order to avoid any resource leaks.
2017-10-13 22:22:03 +03:00
Marko Mäkelä
1b478a7aba MDEV-13311 Presence of old logs in 10.2.7 will corrupt restored instance (change in behavior)
Mariabackup 10.2.7 would delete the redo log files after a successful
--prepare operation. If the user is manually copying the prepared files
instead of using the --copy-back option, it could happen that some old
redo log file would be preserved in the restored location. These old
redo log files could cause corruption of the restored data files when
the server is started up.

We prevent this scenario by creating a "poisoned" redo log file
ib_logfile0 at the end of the --prepare step. The poisoning consists
of simply truncating the file to an empty file. InnoDB will refuse
to start up on an empty redo log file.

copy_back(): Delete all redo log files in the target if the source
file ib_logfile0 is empty. (Previously we did this if the source
file is missing.)

SRV_OPERATION_RESTORE_EXPORT: A new variant of SRV_OPERATION_RESTORE
when the --export option is specified. In this mode, we will keep
deleting all redo log files, instead of truncating the first one.

delete_log_files(): Add a parameter for the first file to delete,
to be passed as 0 or 1.

innobase_start_or_create_for_mysql(): In mariabackup --prepare,
tolerate an empty ib_logfile0 file. Otherwise, require the first
redo log file to be longer than 4 blocks (2048 bytes). Unless
--export was specified, truncate the first log file at the
end of --prepare.
2017-10-10 15:54:11 +03:00