Commit graph

2042 commits

Author SHA1 Message Date
Marko Mäkelä
a9a4089175 Plug a small memory leak in mariabackup --backup 2017-12-14 13:47:38 +02:00
Vesa Pentti
99bcec295d MDEV-12501 -- set --maturity-level by default
* Note: breaking change; since this commit, a plugin that has
    worked so far might get rejected due to plugin maturity
  * mariabackup is not affected (allows all plugins)
  * VERSION file defines SERVER_MATURITY, which defines the
    corresponding numeric value as SERVER_MATURITY_LEVEL in
    include/mysql_version.h
  * The default value for 'plugin_maturity' is SERVER_MATURITY_LEVEL - 1
  * Logs a warning if a plugin has maturity lower than
    SERVER_MATURITY_LEVEL
  * Tests suppress the plugin maturity warning
  * Tests use --plugin-maturity=unknown by default so as not to fail
    due to the stricter plugin maturity handling
2017-12-09 23:34:43 +00:00
Marko Mäkelä
976f6fb1b6 Merge bb-10.2-ext into 10.3 2017-12-06 19:36:33 +02:00
Marko Mäkelä
ce07676502 Merge 10.2 into bb-10.2-ext 2017-12-06 19:34:03 +02:00
Vesa Pentti
5868a184fa Revert "MDEV-12501 -- set --maturity-level by default"
This reverts commit 1af2d7ba23.
2017-12-05 08:49:28 +00:00
Vesa Pentti
1af2d7ba23 MDEV-12501 -- set --maturity-level by default
* Note: breaking change; since this commit, a plugin that has
    worked so far might get rejected due to plugin maturity
  * mariabackup is not affected (allows all plugins)
  * VERSION file defines SERVER_MATURITY, which defines the
    corresponding numeric value as SERVER_MATURITY_LEVEL in
    include/mysql_version.h
  * The default value for 'plugin_maturity' is SERVER_MATURITY_LEVEL - 1
  * Logs a warning if a plugin has maturity lower than
    SERVER_MATURITY_LEVEL
  * Tests suppress the plugin maturity warning
  * Tests use --plugin-maturity=unknown by default so as not to fail
    due to the stricter plugin maturity handling
2017-12-04 21:12:35 +00:00
Monty
60df17e95a Remove compiler warnings 2017-12-03 13:58:36 +02:00
Monty
d9188adae5 resolve_stack_dump updated to match latest stack trace format
Originally only symbols withing [] where resolved. Now we resolve
symbols also withing (+...)

To make it easier to see where the resolved symbol comes from, we
resolve the symbol 'in place' instead of just printing the resolved
symbol alone.
2017-12-03 12:45:54 +02:00
Marko Mäkelä
ddac2d7a1e Merge bb-10.2-ext into 10.3 2017-12-01 15:37:30 +02:00
Marko Mäkelä
24c9785a67 Merge 10.2 into bb-10.2-ext 2017-12-01 15:35:16 +02:00
Vladislav Vaintroub
f59a1826f8 MDEV-14536 : during backup, retry read of log blocks, if there is
(possibly intermittent) checksum mismatch.
2017-11-30 16:30:40 +00:00
Marko Mäkelä
7cb3520c06 Merge bb-10.2-ext into 10.3 2017-11-30 08:16:37 +02:00
Vladislav Vaintroub
bf6d11c4d6 MDEV-14536 : In mariabackup, reread redo log blocks , if checksum mismatch
is detected.

The checksum mismatch can be due to partial write, thus retry the read
2017-11-29 14:53:12 +00:00
Alexander Barkov
5b697c5a23 Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext 2017-11-29 12:06:48 +04:00
Marko Mäkelä
23d2dae5f0 Fix some integer type mismatch warnings 2017-11-28 18:29:20 +02:00
Marko Mäkelä
447cd7b1af Test MDL with a more generic ALTER TABLE statement 2017-11-27 11:20:04 +02:00
Vladislav Vaintroub
414d3a3e17 Fix warning. 2017-11-24 17:33:53 +00:00
Sergey Vojtovich
4cc20c88df Fixed build failure with PFS disabled
mariabackup fails to builds with PFS disabled. The reason was missing include,
which was included by PFS otherwise.
2017-11-24 17:30:40 +00:00
Marko Mäkelä
f1cc6e3874 Merge 10.1 into 10.2 2017-11-24 17:17:16 +02:00
Marko Mäkelä
6979d20426 MDEV-14499 Mariabackup 10.2 fails to back up a multi-file InnoDB system tablespace
When Mariabackup is invoked on an instance that uses a multi-file
InnoDB system tablespace, it may fail to other files of the system
tablespace than the first one.
This was revealed by the MDEV-14447 test case.
The offending code is assuming that the first page of each data file
is page 0. But, in multi-file system tablespaces that is not the case.

xb_fil_cur_open(): Instead of re-reading the first page of the file,
rely on the fil_space_t metadata that already exists in memory.

xb_get_space_flags(): Remove.
2017-11-24 17:12:38 +02:00
Sergey Vojtovich
abf61fd91f Fixed build failure with PFS disabled
mariabackup fails to builds with PFS disabled. The reason was missing include,
which was included by PFS otherwise.
2017-11-24 15:01:55 +04:00
Vladislav Vaintroub
316f0d8fe3 MDEV-14447 mariabackup incremental incorrectly extends system tablespace
for multi-file innodb_data_file_path.

Use fil_extend_space_to_desired_size() to correctly extend system
tablespace. Make sure to get tablespace size from the first tablespace
part.
2017-11-24 00:01:29 +00:00
Sergei Golubchik
7f1900705b Merge branch '10.1' into 10.2 2017-11-21 19:47:46 +01:00
Vladislav Vaintroub
7c4f859384 MDEV-14283 : Fix compilation of mariabackup for gcc3.x 2017-11-21 17:19:32 +00:00
Alexander Barkov
4a8039b04e Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext 2017-11-20 11:12:08 +04:00
Marko Mäkelä
c19ef508b8 InnoDB: Remove ut_snprintf() and the use of my_snprintf(); use snprintf() 2017-11-13 02:11:48 +02:00
Marko Mäkelä
a48aa0cd56 Merge bb-10.2-ext into 10.3 2017-11-10 16:12:45 +02:00
Monty
0bb0d52221 Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext
Conflicts:
	mysql-test/r/cte_recursive.result
	mysql-test/r/derived_cond_pushdown.result
	mysql-test/t/cte_recursive.test
	mysql-test/t/derived_cond_pushdown.test
	sql/datadict.cc
	sql/handler.cc
2017-11-09 23:21:41 +02:00
Vladislav Vaintroub
53c7aaf332 MDEV-14077 Incremental backup extremly slow
Remove the main bottleneck -  the memset() call mentioned in the bug.
Use os_mem_alloc_large() instead of malloc()/memset().
2017-11-09 19:07:05 +00:00
Marko Mäkelä
7c85a8d936 Merge 10.1 into 10.2 2017-11-08 13:12:11 +02:00
Marko Mäkelä
843e4508c0 Merge 10.1 into 10.2 2017-11-07 23:02:39 +02:00
Vladislav Vaintroub
ffb1eebe05 MDEV-13560 Copy all innodb undo tablespaces from the backup directory to destination.
Do not check or rely on innodb_undo_tablespaces in copy-back,
there is no good reason for that.
2017-11-03 16:02:19 +00:00
Alexander Barkov
835cbbcc7b Merge remote-tracking branch 'origin/bb-10.2-ext' into 10.3
TODO: enable MDEV-13049 optimization for 10.3
2017-10-30 20:47:39 +04:00
Alexander Barkov
003cb2f424 Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext 2017-10-30 16:42:46 +04:00
Marko Mäkelä
cbd0da66e4 Merge 10.1 into 10.2 2017-10-25 17:17:21 +03:00
Marko Mäkelä
909cdafd35 MDEV-13496 Use "mariabackup" rather than "xtrabackup" in console output
Replace the remaining "xtrabackup:" references in the code.
2017-10-25 09:06:45 +03:00
Sergei Golubchik
e0a1c745ec Merge branch '10.1' into 10.2 2017-10-24 14:53:18 +02:00
Vladislav Vaintroub
72407e544e MDEV-13496 Use "mariabackup" rather than "xtrabackup" in console output 2017-10-23 10:37:28 +00:00
Vladislav Vaintroub
125ce6f82f MDEV-14102 restore --remove-original options for mariabackup 2017-10-23 10:30:17 +00:00
Sergei Golubchik
2aa51f528f Various compier warnings
gcc 5.4 and 7.1, Debug and Release builds
2017-10-22 14:51:45 +02:00
Sergei Golubchik
9d2e2d7533 Merge branch '10.0' into 10.1 2017-10-22 13:03:41 +02:00
Sergei Golubchik
da4503e956 Merge branch '5.5' into 10.0 2017-10-18 15:14:39 +02:00
Alexander Barkov
30e7d6709f Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext 2017-10-18 14:11:55 +04:00
Sergei Golubchik
df5f25fa7a Merge branch 'mysql/5.5' into 5.5 2017-10-17 10:18:17 +02:00
Sergei Golubchik
d76f5774fe MDEV-13459 Warnings, when compiling with gcc-7.x
mostly caused by -Wimplicit-fallthrough
2017-10-17 07:37:39 +02:00
Marko Mäkelä
9ee840cd0a mariabackup: Properly call os_thread_exit() with detach=true
There is no call to os_thread_join(), so we should detach the thread
handles in order to avoid any resource leaks.
2017-10-13 22:22:03 +03:00
Marko Mäkelä
1b478a7aba MDEV-13311 Presence of old logs in 10.2.7 will corrupt restored instance (change in behavior)
Mariabackup 10.2.7 would delete the redo log files after a successful
--prepare operation. If the user is manually copying the prepared files
instead of using the --copy-back option, it could happen that some old
redo log file would be preserved in the restored location. These old
redo log files could cause corruption of the restored data files when
the server is started up.

We prevent this scenario by creating a "poisoned" redo log file
ib_logfile0 at the end of the --prepare step. The poisoning consists
of simply truncating the file to an empty file. InnoDB will refuse
to start up on an empty redo log file.

copy_back(): Delete all redo log files in the target if the source
file ib_logfile0 is empty. (Previously we did this if the source
file is missing.)

SRV_OPERATION_RESTORE_EXPORT: A new variant of SRV_OPERATION_RESTORE
when the --export option is specified. In this mode, we will keep
deleting all redo log files, instead of truncating the first one.

delete_log_files(): Add a parameter for the first file to delete,
to be passed as 0 or 1.

innobase_start_or_create_for_mysql(): In mariabackup --prepare,
tolerate an empty ib_logfile0 file. Otherwise, require the first
redo log file to be longer than 4 blocks (2048 bytes). Unless
--export was specified, truncate the first log file at the
end of --prepare.
2017-10-10 15:54:11 +03:00
Vladislav Vaintroub
fe18e6b064 MDEV-13822 mariabackup incremental prepare incorrectly sets file size.
MDEV-13310 Preparing an incremental backup twice can corrupt data
2017-10-10 06:19:50 +00:00
Vladislav Vaintroub
b731a5bcf2 Innodb : Refactor os_file_set_size() to be compatible 10.1
The last parameter to this function is now,"bool is_sparse", like in 10.1
rather than the  unused/useless "bool is_readonly", merged from MySQL 5.7

Like in 10.1, this function now supports sparse files, and efficient
platform specific mechanisms for file extension

os_file_set_size() is now consistenly used in all places where
innodb files are extended.
2017-10-10 06:19:50 +00:00
Vladislav Vaintroub
da05d0276a merge 10.1->10.2
Some innobase/xtrabackup changes around  from 10.1 are null merged
, in partucular using os_set_file_size to extend tablespaces in server
or mariabackup.
They require non-trivial amount of additional  work in 10.2, due to
innobase differences between 10.1 and 10.2
2017-10-07 17:43:26 +00:00
Vladislav Vaintroub
bb3f4fbb59 MDEV-13310 Preparing an incremental backup twice can corrupt data
Remove .delta file after it was successfully applied
2017-10-07 08:30:20 +00:00
Vladislav Vaintroub
8d1fb47e1d MDEV-13798 - fix incorrect alignment of the buffer in incremental backup
This incorrect alignment can later lead to memcpy over buffer boundaries,
and to a crash.
2017-10-07 08:30:20 +00:00
Vladislav Vaintroub
0f8295d7d5 MDEV-13822 mariabackup incremental prepare incorrectly sets file size.
Fix incremental prepare to change file size while applying
delta file, if delta file contains page 0 with the new size.
2017-10-07 08:30:20 +00:00
Marko Mäkelä
2c1067166d Merge bb-10.2-ext into 10.3 2017-10-04 08:24:06 +03:00
Alexander Barkov
8ae8cd6348 Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext 2017-10-02 22:35:13 +04:00
Vladislav Vaintroub
7354dc6773 MDEV-13384 - misc Windows warnings fixed 2017-09-28 17:20:46 +00:00
Vladislav Vaintroub
eba44874ca MDEV-13844 : Fix Windows warnings. Fix DBUG_PRINT.
- Fix win64 pointer truncation warnings
(usually coming from misusing 0x%lx and long cast in DBUG)

- Also fix printf-format warnings

Make the above mentioned warnings fatal.

- fix pthread_join on Windows to set return value.
2017-09-28 17:20:46 +00:00
Marko Mäkelä
e3d44f5d62 Merge bb-10.2-ext into 10.3 2017-09-21 08:12:19 +03:00
Marko Mäkelä
fc3b1a7d2f Merge 10.2 into bb-10.2-ext 2017-09-20 17:47:49 +03:00
Vicențiu Ciorbaru
246d321f67 Merge branch '10.2' of github.com:MariaDB/server into 10.2 2017-09-20 10:33:13 +03:00
Marko Mäkelä
5792b016f1 Fix a typo
The function check_mdl_lock_works() is for debug builds only.
2017-09-20 08:44:40 +03:00
Vicențiu Ciorbaru
22c322c649 Merge branch '10.1' into 10.2 2017-09-19 12:43:02 +03:00
Vicențiu Ciorbaru
ec6042bda0 Merge branch '10.0' into 10.1 2017-09-19 12:06:50 +03:00
Marko Mäkelä
e17a282da9 Merge bb-10.2-ext into 10.3 2017-09-18 11:38:07 +03:00
Sergei Golubchik
bba169b984 MDEV-12763 10.2 uses deprecated openssl 1.0 apis even with 1.1
Use OpenSSL 1.1 when applicable.
Create compatibility macros for OpenSSL 1.0- and YaSSL.
2017-09-18 10:12:23 +02:00
Marko Mäkelä
fe949504f0 Merge 10.2 into bb-10.2-ext 2017-09-17 14:03:51 +03:00
Vladislav Vaintroub
d1253e19a1 Fix compilation in mariabackup
Compilation got confused  about 2 wsrep.h headers in include path
Rename backup's wsrep.h to backup_wsrep.h to fixO
2017-09-16 22:19:16 +02:00
Vladislav Vaintroub
ad17e8e518 MDEV-13821 : mariabackup sometimes could lose ib_logf(FATAL) messages,
The messages are getting lost because they are written with fprintf()
but without fflush(), so abort() would lose buffered text.

Applied fix from 10.2, which makes in_logf() use
sql_print_information(), which in turn does proper flush after each message
2017-09-16 09:45:38 +00:00
Vladislav Vaintroub
74f677fcc2 MDEV-13802 mariabackup --lock-ddl-per-table fails when table names contain backticks
use ut_get_name() for formatting database/table names.
2017-09-15 15:49:53 +00:00
Vladislav Vaintroub
bb7ab40521 mariabackup : Fix enumerate_ibd_files() to include .isl 2017-09-15 15:49:53 +00:00
Marko Mäkelä
348eaf4252 Merge bb-10.2-ext into 10.3 2017-09-14 09:12:47 +03:00
Marko Mäkelä
2bd6ccae72 Merge 10.2 into bb-10.2-ext 2017-09-14 09:00:10 +03:00
Marko Mäkelä
3f17f51132 Follow-up to MDEV-13563 mariabackup --lock-ddl-per-table
Fix memory leaks, and add a missing newline to a message.
2017-09-14 08:58:31 +03:00
Alexander Barkov
178540b904 Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext 2017-09-13 12:12:05 +04:00
Marko Mäkelä
66a09bd6ab MDEV-13318 Crash recovery failure after the server is killed during innodb_encrypt_log startup
This fixes several InnoDB bugs related to innodb_encrypt_log and
two Mariabackup --backup bugs.

log_crypt(): Properly derive the initialization vector from the
start LSN of each block. Add a debug assertion.

log_crypt_init(): Note that the function should only be used when
creating redo log files and that the information is persisted in
the checkpoint pages.

xtrabackup_copy_log(): Validate data_len.

xtrabackup_backup_func(): Always use the chosen checkpoint buffer.

log_group_write_buf(), log_write_up_to(): Only log_crypt() the redo
log payload, not the padding bytes.

innobase_start_or_create_for_mysql(): Do not invoke log_crypt_init()
or initiate a redo log checkpoint.

recv_find_max_checkpoint(): Return the contents of LOG_CHECKPOINT_NO
to xtrabackup_backup_func() in log_sys->next_checkpoint_no.
2017-09-12 11:32:49 +03:00
Vladislav Vaintroub
31774f0ede MDEV-13563 lock DDL for mariabackup in 10.2
Implement lock-ddl-per-table option that locks tables before it
is copied to backup, and helds the lock until backup finished

The "DDL-lock" itself is implemented as "SELECT * from <table> LIMIT 0",
inside a transaction, and "COMMIT" of this transaction is the DDL-unlock.
2017-09-12 05:57:05 +00:00
Vladislav Vaintroub
a46679dcf7 Build improvements and cleanups.
- the probably ultimate fix for dependencies on VS
- remove some GET_TARGET_PROPERTY(LOCATION ...), they are deprecated in
cmake 3.9
- simplify signing targets on Windows.
- remove INSTALL_DEBUG_TARGET, we do not mix binaries from different builds
in the same package
2017-09-08 18:22:15 +00:00
Marko Mäkelä
474f51711b Merge bb-10.2-ext into 10.3 2017-09-08 15:59:06 +03:00
Marko Mäkelä
fb14761d66 Merge 10.2 into bb-10.2-ext 2017-09-08 15:44:34 +03:00
Vladislav Vaintroub
18e17f134c Windows : Fix MTR's misuse of servers --console parameter 2017-09-08 08:33:41 +02:00
Vladislav Vaintroub
d471469bd2 MDEV-13466 Implement --export option for MariaDB Backup
full server recovery is performed .
We start "mysqld" with  --bootstrap
and pass bootstrap script consisting of several FLUSH TABLES FOR export/
UNLOCK TABLES
2017-09-07 22:54:06 +00:00
Marko Mäkelä
ae02407ce3 Merge bb-10.2-ext into 10.3 2017-09-07 12:30:56 +03:00
Marko Mäkelä
a82ed92a6a Merge 10.2 into bb-10.2-ext 2017-09-07 12:23:58 +03:00
Marko Mäkelä
b429e8cada Mariabackup: Detach the threads at exit 2017-09-06 19:10:12 +03:00
Marko Mäkelä
4e1fa7f63d Merge bb-10.2-ext into 10.3 2017-09-01 11:33:45 +03:00
Marko Mäkelä
6749d39a95 Merge 10.2 into bb-10.2-ext 2017-09-01 08:47:55 +03:00
Jan Lindström
eca238aea7 MDEV-13557: Startup failure, unable to decrypt ibdata1
Fixes also MDEV-13488: InnoDB writes CRYPT_INFO even though
encryption is not enabled.

Fixes also MDEV-13093: Leak of Datafile::m_crypt_info on
shutdown after failed startup.

Problem was that we created encryption metadata (crypt_data) for
system tablespace even when no encryption was enabled and too early.
System tablespace can be encrypted only using key rotation.

Test innodb-key-rotation-disable, innodb_encryption, innodb_lotoftables
require adjustment because INFORMATION_SCHEMA INNODB_TABLESPACES_ENCRYPTION
contain row only if tablespace really has encryption metadata.

xb_load_single_table_tablespace(): Do not call
fil_space_destroy_crypt_data() any more, because Datafile::m_crypt_data
has been removed.

fil_crypt_realloc_iops(): Avoid divide by zero.

fil_crypt_set_thread_cnt(): Set fil_crypt_threads_event if
encryption threads exist. This is required to find tablespaces
requiring key rotation if no other changes happen.

fil_crypt_find_space_to_rotate(): Decrease the amount of time waiting
when nothing happens to better enable key rotation on startup.

fil_ibd_open(), fil_ibd_load(): Load possible crypt_data from first
page.

class Datafile, class SysTablespace : remove m_crypt_info field.

Datafile::get_first_page(): Return a pointer to first page buffer.

fsp_header_init(): Write encryption metadata to page 0 only if
tablespace is encrypted or encryption is disabled by table option.

i_s_dict_fill_tablespaces_encryption(): Skip tablespaces that do not
contain encryption metadata. This is required to avoid too early
wait condition trigger in encrypted -> unencrypted state transfer.
2017-08-31 08:36:56 +03:00
Marko Mäkelä
f192b48d62 Merge 10.1 into 10.2 2017-08-29 10:07:33 +03:00
Sergei Golubchik
bb8e99fdc3 Merge branch 'bb-10.2-ext' into 10.3 2017-08-26 00:34:43 +02:00
Sergei Golubchik
27412877db Merge branch '10.2' into bb-10.2-ext 2017-08-25 10:25:48 +02:00
Arun Kuruvila
f2f6025a44 Bug#26482173: TLS CIPHER NEGOTIATION INCORRECTLY MATCHES ON
LAST BYTE ONLY (YASSL)


Description:- TLS cipher negociation happens incorrectly
leading to the use of a different

Analysis:- YaSSL based MySQL server will compare only the
last byte of each cipher sent in the Client Hello message.
This can cause TLS connections to fail, due to the server
picking a cipher which the client doesn't actually support.

Fix:- A fix for detecting cipher suites with non leading
zeros is included as YaSSL only supports cipher suites with
leading zeros.
2017-08-24 14:19:38 +05:30
Michael Widenius
458d5ed8aa Lots of small cleanups
- Simplified use_trans_cache() to return at once if is_transactional is set
- Indentation and spelling errors fixed
- Don't call signal_update() if update_binlog_end_pos() is called as the
  function already calls signal_update()
- Removed not used function wait_for_update_bin_log(), which would cause
  errors if ever used.
- Simplified handler::clone() by always allocating 'ref' in ha_open(). To do
  this I added an optional MEM_ROOT argument to ha_open() to be used when
  allocating 'ref'
- Changed arguments to get_system_var() from LEX_CSTRING to LEX_CSTRING*
- Added THD as argument to create_select_for_variable(). Changed also char*
  argument to LEX_CSTRING to avoid strlen() call.
- Change calls to append() to use LEX_CSTRING
2017-08-24 01:05:49 +02:00
Sergei Golubchik
d5a6bae7c0 Fix compilation errors 2017-08-24 01:05:48 +02:00
Sergei Golubchik
4040a17ea2 Compile mariabackup with its own copy of net_serv.cc
Don't use the server's version, that expects a valid THD.
Modify net_serv.cc not not use any THD if MYSQL_SERVER isn't defined.

This reverts commit aaddac5cd7.
2017-08-23 19:05:13 +02:00
Vladislav Vaintroub
aaddac5cd7 fix compile errors 2017-08-23 08:27:46 +00:00
Vladislav Vaintroub
9af7561eb4 MDEV-13608 : set client plugin directory with mysql_options()
if plugin_dir is specified. Also, allow to specify protocol (e.g pipe)
2017-08-21 17:16:12 +00:00
Marko Mäkelä
8a3e2970ad MDEV-13575 On failure, Mariabackup --backup --safe-slave-backup may forget to START SLAVE SQL_THREAD
backup_release(): New function, refactored from backup_finish().
Release some resources that may have been acquired by backup_startup()
and should be released even after a failed operation.

xtrabackup_backup_low(): Refactored from xtrabackup_backup_func().

xtrabackup_backup_func(): Always call backup_release() after calling
backup_start().
2017-08-18 21:42:33 +03:00
Marko Mäkelä
605b835220 MDEV-13754 Memory leak in mariabackup.incremental_backup
The test mariabackup.incremental_backup revealed a memory leak
in have_queries_to_wait_for(). The problem is that
xb_mysql_query() is being invoked with bool use_result=true
but the result is not being freed by mysql_store_result().
There are similar leaks in other functions.

have_queries_to_wait_for(): Invoke mysql_free_result() to
clean up after the mysql_store_result() that was invoked
by xb_mysql_query().

select_incremental_lsn_from_history(): Plug the leak on failure.

kill_long_queries(): Plug the memory leak.
(This function always leaked memory when it was called.)
2017-08-18 10:31:09 +03:00
Marko Mäkelä
74ce0cf148 MDEV-13574 related Mariabackup code cleanup (non-functional change)
have_queries_to_wait_for(), kill_long_queries(): Declare and initialize
variables in one go.
2017-08-18 10:18:26 +03:00
Marko Mäkelä
e9e051d297 Follow-up fix to MDEV-12988 backup fails if innodb_undo_tablespaces>0
The fix broke mariabackup --prepare --incremental.

The restore of an incremental backup starts up (parts of) InnoDB twice.
First, all data files are discovered for applying .delta files. Then,
after the .delta files have been applied, InnoDB will be restarted
more completely, so that the redo log records will be applied via the
buffer pool.

During the first startup, the buffer pool is not initialized, and thus
trx_rseg_get_n_undo_tablespaces() must not be invoked. The apply of
the .delta files will currently assume that the --innodb-undo-tablespaces
option correctly specifies the number of undo tablespace files, just
like --backup does.

The second InnoDB startup of --prepare for applying the redo log will
properly invoke trx_rseg_get_n_undo_tablespaces().

enum srv_operation_mode: Add SRV_OPERATION_RESTORE_DELTA for
distinguishing the apply of .delta files from SRV_OPERATION_RESTORE.

srv_undo_tablespaces_init(): In mariabackup --prepare --incremental,
in the initial SRV_OPERATION_RESTORE_DELTA phase, do not invoke
trx_rseg_get_n_undo_tablespaces() because the buffer pool or the
redo logs are not available. Instead, blindly rely on the parameter
--innodb-undo-tablespaces.
2017-08-18 09:12:04 +03:00
Sergei Golubchik
cb1e76e4de Merge branch '10.1' into 10.2 2017-08-17 11:38:34 +02:00
Marko Mäkelä
5a43c8bae7 MDEV-13416 mariabackup --backup fails to copy log if LSN is above 4294967295
xtrabackup_copy_log(): Use a 64-bit bitmask (lsn_t), not 32-bit (int).
2017-08-11 16:42:27 +03:00
Marko Mäkelä
620ba97cfc Merge remote-tracking branch 'origin/bb-10.2-ext' into 10.3 2017-08-09 12:59:39 +03:00
Marko Mäkelä
1f0a22acbd Merge 10.2 into bb-10.2-ext 2017-08-08 09:47:00 +03:00
Jan Lindström
34eef269eb MDEV-11939: innochecksum mistakes a file for an encrypted one (page 0 invalid)
Always read full page 0 to determine does tablespace contain
encryption metadata. Tablespaces that are page compressed or
page compressed and encrypted do not compare checksum as
it does not exists. For encrypted tables use checksum
verification written for encrypted tables and normal tables
use normal method.

buf_page_is_checksum_valid_crc32
buf_page_is_checksum_valid_innodb
buf_page_is_checksum_valid_none
        Modify Innochecksum logging to file to avoid compilation
	warnings.

fil0crypt.cc fil0crypt.h
        Modify to be able to use in innochecksum compilation and
        move fil_space_verify_crypt_checksum to end of the file.
        Add innochecksum logging to file.

univ.i
        Add innochecksum strict_verify, log_file and cur_page_num
        variables as extern.

page_zip_verify_checksum
        Add innochecksum logging to file and remove unnecessary code.

innochecksum.cc
        Lot of changes most notable able to read encryption
        metadata from page 0 of the tablespace.

Added test case where we corrupt intentionally
FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION (encryption key version)
FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION+4 (post encryption checksum)
FIL_DATA+10 (data)
2017-08-08 09:41:09 +03:00
Jan Lindström
2ef7a5a13a MDEV-13443: Port innochecksum tests from 10.2 innodb_zip suite to 10.1
This is basically port of WL6045:Improve Innochecksum with some
code refactoring on innochecksum.

Added page0size.h include from 10.2 to make 10.1 vrs 10.2 innochecksum
as identical as possible.

Added page 0 checksum checking and if that fails whole test fails.
2017-08-07 12:39:38 +03:00
Monty
19f2b3d02f Fixed compiler warnings 2017-08-07 03:48:58 +03:00
Jan Lindström
8b019f87dd MDEV-11939: innochecksum mistakes a file for an encrypted one (page 0 invalid)
Always read full page 0 to determine does tablespace contain
encryption metadata. Tablespaces that are page compressed or
page compressed and encrypted do not compare checksum as
it does not exists. For encrypted tables use checksum
verification written for encrypted tables and normal tables
use normal method.

buf_page_is_checksum_valid_crc32
buf_page_is_checksum_valid_innodb
buf_page_is_checksum_valid_none
	Add Innochecksum logging to file

buf_page_is_corrupted
        Remove ib_logf and page_warn_strict_checksum
        calls in innochecksum compilation. Add innochecksum
        logging to file.

fil0crypt.cc fil0crypt.h
        Modify to be able to use in innochecksum compilation and
	move fil_space_verify_crypt_checksum to end of the file.
	Add innochecksum logging to file.

univ.i
        Add innochecksum strict_verify, log_file and cur_page_num
        variables as extern.

page_zip_verify_checksum
        Add innochecksum logging to file.

innochecksum.cc
        Lot of changes most notable able to read encryption
        metadata from page 0 of the tablespace.

Added test case where we corrupt intentionally
FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION (encryption key version)
FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION+4 (post encryption checksum)
FIL_DATA+10 (data)
2017-08-03 08:29:36 +03:00
Alexander Barkov
e67b816451 Merge remote-tracking branch 'origin/bb-10.2-ext' into 10.3 2017-07-19 19:43:55 +04:00
Alexander Barkov
34668e10b2 Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext 2017-07-19 14:45:54 +04:00
Vladislav Vaintroub
013595f56f MDEV-13332 mariabackup from 10.2.x crashes with --ftwrl-* options
Fixed null pointer dereference in parsing "show full processlist" output
with atoi().

Some  Innodb background thread has NULL in 'Time' column,
thus  backup would crash with when atoi is applied to null pointer.
2017-07-17 17:05:06 +00:00
Vladislav Vaintroub
f58142f644 Mariabackup : don't change argv[0] to "innobackupex" in innobackupex mode.
addr2line utility optionally used to output stacktrace relies relies on
correct my_progname, which is initialized from argv[0] from main function.
Thus, changing argv[0] can confuse stacktrace output.
2017-07-13 14:49:57 +00:00
Marko Mäkelä
57fea53615 Merge bb-10.2-ext into 10.3 2017-07-07 12:39:43 +03:00
Marko Mäkelä
9e53a6bdfd Merge 10.2 into bb-10.2-ext 2017-07-07 12:00:27 +03:00
Marko Mäkelä
f20693c231 Remove a reference to a non-existent include directory 2017-07-07 11:57:39 +03:00
Alexander Barkov
8b2c7c9444 Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext 2017-07-07 12:43:10 +04:00
Marko Mäkelä
99d52c45cb Mariabackup: Copy all of the redo log correctly
xtrabackup_copy_log(), xtrabackup_copy_logfile():
Change the Boolean parameter to an enum, with the values
COPY_FIRST, COPY_ONLINE, COPY_LAST.

xtrabackup_copy_log(): Return the latest scanned LSN,
which may be less than the last copied LSN. Remove some
dead code that was duplicating some logic that in 10.2
has been moved to log_group_read_log_seg().

log_copying_thread(): Correct the termination condition.

stop_backup_threads(): Shut down the threads that were
created during backup.
2017-07-06 20:17:46 +03:00
Marko Mäkelä
d7b21a49c2 Mariabackup: Remove unused parameters and fix some memory leaks
Use GET_STR instead of GET_STR_ALLOC, so that the memory will
cannot be leaked. For some reason, calling my_cleanup_options()
on xb_server_options or xb_client_options would not work.
2017-07-06 19:14:44 +03:00
Sergei Golubchik
f6633bf058 Merge branch '10.1' into 10.2 2017-07-05 19:08:55 +02:00
Marko Mäkelä
e3d3147792 MDEV-13105 InnoDB fails to load a table with PAGE_COMPRESSION_LEVEL after upgrade from 10.1.20
When using innodb_page_size=16k, InnoDB tables
that were created in MariaDB 10.1.0 to 10.1.20 with
PAGE_COMPRESSED=1 and
PAGE_COMPRESSION_LEVEL=2 or PAGE_COMPRESSION_LEVEL=3
would fail to load.

fsp_flags_is_valid(): When using innodb_page_size=16k, use a
more strict check for .ibd files, with the assumption that
nobody would try to use different-page-size files.
2017-07-05 14:35:55 +03:00
Marko Mäkelä
8c71c6aa8b MDEV-12548 Initial implementation of Mariabackup for MariaDB 10.2
InnoDB I/O and buffer pool interfaces and the redo log format
have been changed between MariaDB 10.1 and 10.2, and the backup
code has to be adjusted accordingly.

The code has been simplified, and many memory leaks have been fixed.
Instead of the file name xtrabackup_logfile, the file name ib_logfile0
is being used for the copy of the redo log. Unnecessary InnoDB startup and
shutdown and some unnecessary threads have been removed.

Some help was provided by Vladislav Vaintroub.

Parameters have been cleaned up and aligned with those of MariaDB 10.2.

The --dbug option has been added, so that in debug builds,
--dbug=d,ib_log can be specified to enable diagnostic messages
for processing redo log entries.

By default, innodb_doublewrite=OFF, so that --prepare works faster.
If more crash-safety for --prepare is needed, double buffering
can be enabled.

The parameter innodb_log_checksums=OFF can be used to ignore redo log
checksums in --backup.

Some messages have been cleaned up.
Unless --export is specified, Mariabackup will not deal with undo log.
The InnoDB mini-transaction redo log is not only about user-level
transactions; it is actually about mini-transactions. To avoid confusion,
call it the redo log, not transaction log.

We disable any undo log processing in --prepare.

Because MariaDB 10.2 supports indexed virtual columns, the
undo log processing would need to be able to evaluate virtual column
expressions. To reduce the amount of code dependencies, we will not
process any undo log in prepare.

This means that the --export option must be disabled for now.

This also means that the following options are redundant
and have been removed:
	xtrabackup --apply-log-only
	innobackupex --redo-only

In addition to disabling any undo log processing, we will disable any
further changes to data pages during --prepare, including the change
buffer merge. This means that restoring incremental backups should
reliably work even when change buffering is being used on the server.
Because of this, preparing a backup will not generate any further
redo log, and the redo log file can be safely deleted. (If the
--export option is enabled in the future, it must generate redo log
when processing undo logs and buffered changes.)

In --prepare, we cannot easily know if a partial backup was used,
especially when restoring a series of incremental backups. So, we
simply warn about any missing files, and ignore the redo log for them.

FIXME: Enable the --export option.

FIXME: Improve the handling of the MLOG_INDEX_LOAD record, and write
a test that initiates a backup while an ALGORITHM=INPLACE operation
is creating indexes or rebuilding a table. An error should be detected
when preparing the backup.

FIXME: In --incremental --prepare, xtrabackup_apply_delta() should
ensure that if FSP_SIZE is modified, the file size will be adjusted
accordingly.
2017-07-05 11:43:28 +03:00
Marko Mäkelä
4fe89773d8 Mariabackup: Clean up xtrabackup options
Fix the binding of databases_file. It was incorrectly mapped
to OPT_XTRA_TABLES_FILE.

Remove some unused options and variables.
2017-06-30 11:35:28 +03:00
Vladislav Vaintroub
ba9daddcd8 Cherrypick Perconas fix for leaking descriptors with new xbstream.
72c55f4fc2
2017-06-22 12:05:36 +00:00
Marko Mäkelä
2a3fe45dd2 Remove XtraDB
The XtraDB storage engine was already replaced by InnoDB
and disabled in MariaDB Server 10.2. Let us remove it altogether
to avoid dragging dead code around.

Replace some references to XtraDB with references to InnoDB.

rpl_get_position_info(): Remove.

Remove the mysql-test-run --suite=percona, because it only contains
tests specific to XtraDB, many of which were disabled already in
earlier MariaDB versions.
2017-06-21 13:44:16 +03:00
Vladislav Vaintroub
b9a326b6e1 MDEV-12709 : mariabackup - during backup phase read some innodb parameter
using "show variables", rather than take the value from my.cnf.

"show variables" is more accurate than my.cnf,it also works for parameters
set on the mysqld command line, which is especially important for MTR.
2017-06-19 17:20:30 +00:00
Marko Mäkelä
1e3886ae80 Merge bb-10.2-ext into 10.3 2017-06-19 17:28:08 +03:00
Marko Mäkelä
3a7201ea92 Merge 10.2 into bb-10.2-ext 2017-06-19 16:56:13 +03:00
Marko Mäkelä
c73fa2d75f Merge 10.1 into 10.2
This will also change the minimum and maximum value of
innodb_log_file_size to 1MiB and 512GiB, respectively.
2017-06-19 16:46:34 +03:00
Marko Mäkelä
9a646c91dd Mariabackup: Remove the --stats option
The option was basically duplicating InnoDB functionality.
Persistent statistics can be accessed via the tables
mysql.innodb_table_stats and mysql.innodb_index_stats.
2017-06-19 11:27:49 +03:00
Marko Mäkelä
cede2b6f0f Mariabackup: Remove support for .xbcrypt files 2017-06-19 11:27:49 +03:00
Marko Mäkelä
7e22050e66 Mariabackup: Remove the options --compact --rebuild-indexes --rebuild-threads 2017-06-19 11:27:27 +03:00
Marko Mäkelä
fa70d077f7 Mariabackup: Remove the options --to-archived-lsn --innodb-log-arch-dir
These functions cannot possibly work in MariaDB 10.2, and it is
uncertain if they really work in 10.1 either.
2017-06-19 11:01:44 +03:00
Alexander Barkov
765347384a Merge remote-tracking branch 'origin/10.2' into bb-10.2-ext 2017-06-15 15:27:11 +04:00
Marko Mäkelä
a78476d342 Merge 10.1 into 10.2 2017-06-12 17:43:07 +03:00
Marko Mäkelä
fa57479fcd Merge 10.0 into 10.1 2017-06-12 14:26:32 +03:00
Marko Mäkelä
2d8fdfbde5 Merge 10.1 into 10.2
Replace have_innodb_zip.inc with innodb_page_size_small.inc.
2017-06-08 12:45:08 +03:00
Marko Mäkelä
86927cc712 Remove traces of multiple InnoDB redo logs
InnoDB never supported more than one copy of a redo log.
There were provisions to do that. For Mariabackup, let us clean up
this code.

log_sys_init(): Renamed from log_init().

log_set_capacity(): Renamed from log_calc_max_ages().

log_init(): Renamed from log_group_init(). Remove the parameters
id, space_id. Let the caller invoke log_set_capacity() when needed.

log_group_t: Remove id, space_id, log_groups.

log_t: Replace log_groups with a single log.

recv_find_max_checkpoint(): Declare globally. Remove the first parameter.

xtrabackup_choose_lsn_offset(): Remove (dead code).
2017-06-06 12:06:41 +03:00
Marko Mäkelä
0c92794db3 Remove deprecated InnoDB file format parameters
The following options will be removed:

innodb_file_format
innodb_file_format_check
innodb_file_format_max
innodb_large_prefix

They have been deprecated in MySQL 5.7.7 (and MariaDB 10.2.2) in WL#7703.

The file_format column in two INFORMATION_SCHEMA tables will be removed:

innodb_sys_tablespaces
innodb_sys_tables

Code to update the file format tag at the end of page 0:5
(TRX_SYS_PAGE in the InnoDB system tablespace) will be removed.
When initializing a new database, the bytes will remain 0.

All references to the Barracuda file format will be removed.
Some references to the Antelope file format (meaning
ROW_FORMAT=REDUNDANT or ROW_FORMAT=COMPACT) will remain.

This basically ports WL#7704 from MySQL 8.0.0 to MariaDB 10.3.1:

commit 4a69dc2a95995501ed92d59a1de74414a38540c6
Author: Marko Mäkelä <marko.makela@oracle.com>
Date:   Wed Mar 11 22:19:49 2015 +0200
2017-06-02 09:36:14 +03:00
Jan Lindström
1af8bf39ca MDEV-12113: install_db shows corruption for rest encryption with innodb_data_file_path=ibdata1:3M;
Problem was that FIL_PAGE_FLUSH_LSN_OR_KEY_VERSION field that for
encrypted pages even in system datafiles should contain key_version
except very first page (0:0) is after encryption overwritten with
flush lsn.

Ported WL#7990 Repurpose FIL_PAGE_FLUSH_LSN to 10.1
The field FIL_PAGE_FLUSH_LSN_OR_KEY_VERSION is consulted during
InnoDB startup.

At startup, InnoDB reads the FIL_PAGE_FLUSH_LSN_OR_KEY_VERSION
from the first page of each file in the InnoDB system tablespace.
If there are multiple files, the minimum and maximum LSN can differ.
These numbers are passed to InnoDB startup.

Having the number in other files than the first file of the InnoDB
system tablespace is not providing much additional value. It is
conflicting with other use of the field, such as on InnoDB R-tree
index pages and encryption key_version.

This worklog will stop writing FIL_PAGE_FLUSH_LSN_OR_KEY_VERSION to
other files than the first file of the InnoDB system tablespace
(page number 0:0) when system tablespace is encrypted. If tablespace
is not encrypted we continue writing FIL_PAGE_FLUSH_LSN_OR_KEY_VERSION
to all first pages of system tablespace to avoid unnecessary
warnings on downgrade.

open_or_create_data_files(): pass only one flushed_lsn parameter

xb_load_tablespaces(): pass only one flushed_lsn parameter.

buf_page_create(): Improve comment about where
FIL_PAGE_FIL_FLUSH_LSN_OR_KEY_VERSION is set.

fil_write_flushed_lsn(): A new function, merged from
fil_write_lsn_and_arch_no_to_file() and
fil_write_flushed_lsn_to_data_files().
Only write to the first page of the system tablespace (page 0:0)
if tablespace is encrypted, or write all first pages of system
tablespace and invoke fil_flush_file_spaces(FIL_TYPE_TABLESPACE)
afterwards.

fil_read_first_page(): read flush_lsn and crypt_data only from
first datafile.

fil_open_single_table_tablespace(): Remove output of LSN, because it
was only valid for the system tablespace and the undo tablespaces, not
user tablespaces.

fil_validate_single_table_tablespace(): Remove output of LSN.

checkpoint_now_set(): Use fil_write_flushed_lsn and output
a error if operation fails.

Remove lsn variable from fsp_open_info.

recv_recovery_from_checkpoint_start(): Remove unnecessary second
flush_lsn parameter.

log_empty_and_mark_files_at_shutdown(): Use fil_writte_flushed_lsn
and output error if it fails.

open_or_create_data_files(): Pass only one flushed_lsn variable.
2017-06-01 14:07:48 +03:00
Marko Mäkelä
acea8b5bad Fix some integer type mismatch in innochecksum 2017-05-26 22:45:53 +03:00
Alexander Barkov
9bc3225642 Merge tag 'mariadb-10.2.6' into bb-10.2-ext 2017-05-26 19:32:28 +04:00
Marko Mäkelä
8f643e2063 Merge 10.1 into 10.2 2017-05-23 11:09:47 +03:00
Marko Mäkelä
70505dd45b Merge 10.1 into 10.2 2017-05-22 09:46:51 +03:00
Vladislav Vaintroub
ee4eda40b9 MDEV-12832 : remove libarchive support from mariabackup,
due to different packaging issues.

Also, Percona thinks that tar support has  many limitations
and should be removed as well( see discussion  in
https://bugs.launchpad.net/percona-xtrabackup/+bug/1681721)

there is an alternative streaming format xbstream that is supported and
does not have these limitations.
2017-05-21 22:19:06 +00:00
Marko Mäkelä
65e1399e64 Merge 10.0 into 10.1
Significantly reduce the amount of InnoDB, XtraDB and Mariabackup
code changes by defining pfs_os_file_t as something that is
transparently compatible with os_file_t.
2017-05-20 08:41:20 +03:00
Marko Mäkelä
13a350ac29 Merge 10.0 into 10.1 2017-05-19 12:29:37 +03:00
Vladislav Vaintroub
9dffa3072c MDEV-12810 - force static build of crc library 2017-05-18 11:01:34 +00:00
Vladislav Vaintroub
40c7778e05 MDEV-12814 mariabackup : don't try io throttling in copy-back
Throttling only works with when creating backup.  Attempt to use it with
 --copy-back results in crash, since throttle events are not initialized.
Thus, ignore throttling unless --backup is given.
2017-05-18 11:01:34 +00:00
Marko Mäkelä
71cd205956 Silence bogus GCC 7 warnings -Wimplicit-fallthrough
Do not silence uncertain cases, or fix any bugs.

The only functional change should be that ha_federated::extra()
is not calling DBUG_PRINT to report an unhandled case for
HA_EXTRA_PREPARE_FOR_DROP.
2017-05-17 08:27:04 +03:00
Marko Mäkelä
7972da8aa1 Silence bogus GCC 7 warnings -Wimplicit-fallthrough
Do not silence uncertain cases, or fix any bugs.

The only functional change should be that ha_federated::extra()
is not calling DBUG_PRINT to report an unhandled case for
HA_EXTRA_PREPARE_FOR_DROP.
2017-05-17 08:07:02 +03:00