Commit graph

435 commits

Author SHA1 Message Date
Vladislav Vaintroub
ed66acb291 Silence LeakSanitizer by default in mariabackup, so that phanthom "leaks"
would not hide more interesting information, like invalid memory accesses.


some "leaks" are expected
- partly this is due to weird options parsing, that runs twice, and
does not free memory after the first run.
- also we do not mind to exit()  whenever it makes sense, without full
cleanup.
2018-12-29 02:06:19 +01:00
Marko Mäkelä
cb2f36d3ea MDEV-14975: Cleanup check_all_privileges()
Remove an unused variable, and propagate the error to the caller
instead of calling exit().
2018-12-28 15:33:34 +02:00
Vladislav Vaintroub
773479f5b3 Add test for partial backup for partitioned table. 2018-12-21 16:04:16 +01:00
Marko Mäkelä
b7a9563b21 Merge 10.1 into 10.2 2018-12-21 09:43:35 +02:00
Vladislav Vaintroub
9f4a4cb401 Cleanup recent mariabackup validation patches.
- Refactor code to isolate page validation in page_is_corrupted() function.

- Introduce --extended-validation parameter(default OFF) for mariabackup
--backup to enable decryption of encrypted uncompressed pages during
backup.

- mariabackup would still always check checksum on encrypted data,
it is needed to detect  partially written pages.
2018-12-20 14:31:18 +01:00
Marko Mäkelä
610e4034d7 Merge 10.1 into 10.2 2018-12-19 15:55:55 +02:00
Marko Mäkelä
dd72d7d561 MDEV-18025: Improve test case and consistency checks
Write a test case that computes valid crc32 checksums for
an encrypted page, but zeroes out the payload area, so
that the checksum after decryption fails.

xb_fil_cur_read(): Validate the page number before trying
any checksum calculation or decrypting or decompression.
Also, skip zero-filled pages. For page_compressed pages,
ensure that the FIL_PAGE_TYPE was changed. Also, reject
FIL_PAGE_PAGE_COMPRESSED_ENCRYPTED if no decryption was attempted.
2018-12-19 15:45:35 +02:00
Vladislav Vaintroub
74659e55b7 MDEV-16268 : sporadic checksum mismatch when opening system tablespace in backup
Attempt to fix by retrying srv_sys_space.open_or_create() if it reports
corruption.
2018-12-18 17:33:42 +01:00
Marko Mäkelä
560df47926 Merge 10.1 into 10.2 2018-12-18 16:28:19 +02:00
Thirunarayanan Balathandayuthapani
171271edf8 MDEV-18025 Mariabackup fails to detect corrupted page_compressed=1 tables
Problem:
=======
Mariabackup seems to fail to verify the pages of compressed tables.
The reason is that both fil_space_verify_crypt_checksum() and
buf_page_is_corrupted() will skip the validation for compressed pages.

Fix:
====
Mariabackup should call fil_page_decompress() for compressed and encrypted
compressed page. After that, call buf_page_is_corrupted() to
check the page corruption.
2018-12-18 18:07:17 +05:30
Marko Mäkelä
b5763ecd01 Merge 10.3 into 10.4 2018-12-18 11:33:53 +02:00
Marko Mäkelä
45531949ae Merge 10.2 into 10.3 2018-12-18 09:15:41 +02:00
Marko Mäkelä
7d245083a4 Merge 10.1 into 10.2 2018-12-17 20:15:38 +02:00
Marko Mäkelä
8c43f96388 Follow-up to MDEV-12112: corruption in encrypted table may be overlooked
The initial fix only covered a part of Mariabackup.
This fix hardens InnoDB and XtraDB in a similar way, in order
to reduce the probability of mistaking a corrupted encrypted page
for a valid unencrypted one.

This is based on work by Thirunarayanan Balathandayuthapani.

fil_space_verify_crypt_checksum(): Assert that key_version!=0.
Let the callers guarantee that. Now that we have this assertion,
we also know that buf_page_is_zeroes() cannot hold.
Also, remove all diagnostic output and related parameters,
and let the relevant callers emit such messages.
Last but not least, validate the post-encryption checksum
according to the innodb_checksum_algorithm (only accepting
one checksum for the strict variants), and no longer
try to validate the page as if it was unencrypted.

buf_page_is_zeroes(): Move to the compilation unit of the only callers,
and declare static.

xb_fil_cur_read(), buf_page_check_corrupt(): Add a condition before
calling fil_space_verify_crypt_checksum(). This is a non-functional
change.

buf_dblwr_process(): Validate the page only as encrypted or unencrypted,
but not both.
2018-12-17 19:33:44 +02:00
Vladislav Vaintroub
0a2edddbf4 MDEV-14975 : fix last commit's typo. 2018-12-15 00:06:00 +01:00
Vladislav Vaintroub
5716c71c54 MDEV-14975 mariabackup starts with unprivileged user.
ported privilege checking from xtrabackup.
Now, mariabackup would terminate early if either RELOAD or PROCESS privilege
is not held, not at the very end of backup

The behavior can be disabled with nre setting --check-privileges=0.
Also , --no-lock does not need all of these privileges, since it skips
FTWRL and SHOW ENGINE STATUS INNODB.
2018-12-14 23:36:21 +01:00
Marko Mäkelä
fd37344feb Merge 10.3 into 10.4 2018-12-14 16:20:17 +02:00
Marko Mäkelä
5fefcb0a21 Merge 10.2 into 10.3 2018-12-14 16:15:59 +02:00
Marko Mäkelä
94fa02f4d0 Merge 10.1 into 10.2 2018-12-14 16:11:05 +02:00
Marko Mäkelä
fb252f70c1 MDEV-12112 corruption in encrypted table may be overlooked
After validating the post-encryption checksum on an encrypted page,
Mariabackup should decrypt the page and validate the pre-encryption
checksum as well. This should reduce the probability of accepting
invalid pages as valid ones.

This is a backport and refactoring of a patch that was
originally written by Thirunarayanan Balathandayuthapani
for the 10.2 branch.
2018-12-14 15:44:51 +02:00
Vladislav Vaintroub
3efed7533c Fix lock_ddl_per_table handling in mariabackup.
mariabackup.lock_ddl_per_table was broken, after MDEV-17772 got fixed.
Fix removes MDL waiter killer, it is is not needed anymore.
2018-12-10 14:03:52 +01:00
Vladislav Vaintroub
c0ca164b1c MDEV-17308 mariabackup to use the BACKUP statement instead of FTWRL 2018-12-09 22:12:27 +02:00
Marko Mäkelä
b374246730 Merge 10.3 into 10.4 2018-11-30 11:05:46 +02:00
Marko Mäkelä
0abd2766b1 Merge 10.2 into 10.3
Also, related to MDEV-15522, MDEV-17304, MDEV-17835,
remove the Galera xtrabackup tests, because xtrabackup never worked
with MariaDB Server 10.3 due to InnoDB redo log format changes.
2018-11-30 09:38:56 +02:00
Marko Mäkelä
447e493179 Remove some unnecessary InnoDB #include 2018-11-29 12:53:44 +02:00
Marko Mäkelä
926b04e550 Merge 10.3 into 10.4 2018-11-28 01:18:30 +02:00
Marko Mäkelä
babb000a36 Merge 10.2 into 10.3 2018-11-28 01:02:46 +02:00
Marko Mäkelä
e82e216e37 MDEV-17849 Undo tablespace truncation recovery fails to shrink file
fil_space_t::add(): Replaces fil_node_create(), fil_node_create_low().
Let the caller pass fil_node_t::handle, to avoid having to close and
re-open files.

fil_node_t::read_page0(): Refactored from fil_node_open_file().
Read the first page of a data file.

fil_node_open_file(): Open the file only once.

srv_undo_tablespace_open(): Set the file handle for the opened
undo tablespace. This should ensure that ut_ad(file->is_open())
no longer fails in recv_add_trim().

xtrabackup_backup_func(): Remove some dead code.

xb_fil_cur_open(): Open files only if needed. Undo tablespaces
should already have been opened.
2018-11-27 14:49:39 +02:00
Marko Mäkelä
eb6364619f Remove the redundant variable fil_n_file_opened 2018-11-27 14:30:39 +02:00
Marko Mäkelä
074c684099 Merge 10.3 into 10.4 2018-11-06 16:24:16 +02:00
Marko Mäkelä
df563e0c03 Merge 10.2 into 10.3
main.derived_cond_pushdown: Move all 10.3 tests to the end,
trim trailing white space, and add an "End of 10.3 tests" marker.
Add --sorted_result to tests where the ordering is not deterministic.

main.win_percentile: Add --sorted_result to tests where the
ordering is no longer deterministic.
2018-11-06 09:40:39 +02:00
Vladislav Vaintroub
f8268f3cce MDEV-17195 Speedup lock-ddl-per-table, if there is a large number of tables
Query INFORMATION_SCHEMA.INNODB_SYS_TABLES only once, and cache results.

As a small cleanup, remove  mdl_lock_con_mutex, the MDL handling
connection is never accessed by multiple threads at the same time.
2018-10-31 17:19:25 +01:00
Sergei Golubchik
44f6f44593 Merge branch '10.0' into 10.1 2018-10-30 15:10:01 +01:00
Daniel Black
3859273d04 MDEV-14267: correct FSF address 2018-10-30 19:45:09 +08:00
Marko Mäkelä
444c380ceb Merge 10.3 into 10.4 2018-10-05 08:09:49 +03:00
Sergei Golubchik
57e0da50bb Merge branch '10.2' into 10.3 2018-09-28 16:37:06 +02:00
Marko Mäkelä
304857764b MDEV-13564 Mariabackup does not work with TRUNCATE
Implement undo tablespace truncation via normal redo logging.

Implement TRUNCATE TABLE as a combination of RENAME to #sql-ib name,
CREATE, and DROP.
2018-09-25 17:29:43 +03:00
Sergei Golubchik
5ae8fce50b Merge branch '10.1' into 10.2 2018-09-24 11:46:08 +02:00
Marko Mäkelä
8b6b2c3ea1 Fix mariabackup leaks (except my_load_defaults) 2018-09-21 11:18:59 +03:00
Vladislav Vaintroub
c139dc6d38 Windows : Fix application verifier errors.
Make different threads that are running data_copy_thread_func()
use the same pthread_mutex for locking (not the copies of the same
pthread_mutex_t).
2018-09-20 19:27:59 +01:00
Vladislav Vaintroub
0fa35ddf1f Amend fix for MDEV-17236
Simplify, and make it work with system tablespace outside of
innodb data home.

Also, do not reread TRX_SYS page  in endless loop,
if it appears to be corrupted.

Use finite number of attempts.
2018-09-20 14:08:57 +01:00
Thirunarayanan Balathandayuthapani
7e4bbd3aa6 MDEV-17236 mariabackup incorrectly tries to open ibdata1
if custom undo tablespace is defined

- In case of multiple undo tablespace, mariabackup have to open system
tablespace to find the list of undo tablespace present in TRX_SYS page.
For opening system tablespace, mariabackup should fetch the file name
from already initialized system tablespace object.
2018-09-20 17:13:12 +05:30
Marko Mäkelä
65474d92f5 Follow-up to "Fixed wrong printf in mariabackup"
This amends commit 4dc20ff687.

Starting with MariaDB 10.2, InnoDB defines
typedef size_t ulint;

The standard format for size_t uses the z modifier, for example,
"%zu" as in the macro ULINTPF.

"%lu" is wrong for size_t, because sizeof(unsigned long) can be
something else than sizeof(size_t). On Windows, the former would
always be 4 bytes, while size_t would be 4 or 8 bytes.
2018-09-18 06:23:25 +03:00
Michael Widenius
4dc20ff687 Fixed wrong printf in mariabackup 2018-09-16 11:23:27 +03:00
Vladislav Vaintroub
6b2da93359 MDEV-17192 Backup with -no-lock should fail, if DDL is detected at the end of backup 2018-09-14 09:36:02 +01:00
Oleksandr Byelkin
28f08d3753 Merge branch '10.1' into 10.2 2018-09-14 08:47:22 +02:00
Vladislav Vaintroub
e46b2a3e94 MDEV-12956 provide default datadir for mariabackup --copy-back
On Unix, it is compiled-in datadir value.
On Windows, the directory is ..\data, relative to directory
mariabackup.exe

server uses the same logic to determine datadir.
2018-09-11 20:59:35 +01:00
Marko Mäkelä
09af00cbde MDEV-13564: Remove old crash-upgrade logic in 10.4
Stop supporting the additional *trunc.log files that were
introduced via MySQL 5.7 to MariaDB Server 10.2 and 10.3.

DB_TABLESPACE_TRUNCATED: Remove.

purge_sys.truncate: A new structure to track undo tablespace
file truncation.

srv_start(): Remove the call to buf_pool_invalidate(). It is
no longer necessary, given that we no longer access things in
ways that violate the ARIES protocol. This call was originally
added for innodb_file_format, and it may later have been necessary
for the proper function of the MySQL 5.7 TRUNCATE recovery, which
we are now removing.

trx_purge_cleanse_purge_queue(): Take the undo tablespace as a
parameter.

trx_purge_truncate_history(): Rewrite everything mostly in a
single function, replacing references to undo::Truncate.

recv_apply_hashed_log_recs(): If any redo log is to be applied,
and if the log_sys.log.subformat indicates that separately
logged truncate may have been used, refuse to proceed except if
innodb_force_recovery is set. We will still refuse crash-upgrade
if TRUNCATE TABLE was logged. Undo tablespace truncation would
only be logged in undo*trunc.log files, which we are no longer
checking for.
2018-09-11 21:32:15 +03:00
Marko Mäkelä
67fa97dc2c Merge 10.3 into 10.4 2018-09-11 21:31:47 +03:00
Marko Mäkelä
1bf3e8ab43 Merge 10.3 into 10.4 2018-09-11 21:31:03 +03:00