Commit graph

901 commits

Author SHA1 Message Date
Marko Mäkelä
0792aff161 Merge 10.4 into 10.5 2022-09-20 13:17:02 +03:00
Marko Mäkelä
0c0a569028 Merge 10.3 into 10.4 2022-09-20 12:38:25 +03:00
Marko Mäkelä
c22dff21a5 InnoDB cleanup: Replace UNIV_LINUX, UNIV_SOLARIS, UNIV_AIX
Let us use the normal platform-specific preprocessor symbols
__linux__, __sun__, _AIX instead of some homebrew ones.

The preprocessor symbol UNIV_HPUX must have lost its meaning
by f6deb00a56 (note: the symbol
UNIV_HPUX10 is being checked for, but only UNIV_HPUX is defined).
2022-09-19 12:20:53 +03:00
Daniel Black
3c8674edcc Merge 10.4 into 10.5 2022-09-19 17:03:17 +10:00
Marko Mäkelä
3e3cfa8934 MDEV-18589 Assertion on info.page_size failed in xb_delta_open_matching_space
xb_read_delta_metadata(): For ROW_FORMAT=COMPRESSED tables, initialize
the info.zip_size with the physical page size and let info.page_size
remain the logical page size, like xb_delta_open_matching_space()
expects it to be ever since
commit 0a1c3477bf (MDEV-18493).
2022-09-14 11:39:30 +03:00
Alexander Barkov
8f9df08f02 MDEV-19246 Change database and table used for Mariabackup's history
Changing the mariabackup history table from PERCONA_SCHEMA.xtrabackup_history
to mysql.mariabackup_history.

Additionally, extending xb_history.test for better coverage:

- Recording the fact that the history table is created during
  "mariabackup --history" invocation when it does not exist.

- Recording the history table structure (adding SHOW CREATE TABLE)

- Recording how --history vs --history=foo affect the "name" column
  of the history table.

- Recording the fact that two consequent executions of
  "mariabackup --history[=foo]" insert into the history table
  incrementally, without truncating it on every execution.
2022-09-12 14:23:38 +04:00
Marko Mäkelä
e71aca8200 Merge 10.9 into 10.10 2022-08-30 13:33:02 +03:00
Marko Mäkelä
c8cd162a0a Merge 10.7 into 10.8 2022-08-30 13:04:17 +03:00
Marko Mäkelä
b86be02ecf Merge 10.6 into 10.7 2022-08-30 13:02:42 +03:00
Marko Mäkelä
f410974f0f Merge 10.5 into 10.6 2022-08-30 13:01:16 +03:00
Marko Mäkelä
259050f864 Merge 10.9 into 10.10 2022-08-29 14:04:25 +03:00
Daniel Black
0324bde846 mariabackup: remove MySQL wording 2022-08-26 11:52:53 +10:00
Daniel Black
79b58f1ca8 MDEV-23607 MariaBackup - align required GRANTS to cmd options
Since the 10.5 split of the privileges, the required GRANTs
for various mariabackup operations has changed.

In the addition of tests, a number of mappings where incorrect:

The option --lock-ddl-per-table didn't require connection admin.

The option --safe-slave-backup requires SLAVE MONITOR even without
the --no-lock option.
2022-08-26 11:52:53 +10:00
Marko Mäkelä
2bddc5d045 Merge 10.7 into 10.8 2022-08-24 10:22:37 +03:00
Marko Mäkelä
bdd80e3fb1 Merge 10.6 into 10.7 2022-08-24 09:22:34 +03:00
Marko Mäkelä
d65a2b7bde Merge 10.5 into 10.6 2022-08-22 14:02:43 +03:00
Marko Mäkelä
1d90d6874d Merge 10.4 into 10.5 2022-08-22 13:38:40 +03:00
Marko Mäkelä
36d173e523 Merge 10.3 into 10.4 2022-08-22 12:34:42 +03:00
Marko Mäkelä
c2df3d30c0 MDEV-21452 fixup: Avoid an unnecessary mutex operation 2022-08-19 09:21:02 +03:00
Marko Mäkelä
a1055ab35d MDEV-29043 mariabackup --compress hangs
Even though commit b817afaa1c passed
the test mariabackup.compress_qpress, that test turned out to be
too small to reveal one more problem that had previously been prevented
by the existence of ctrl_mutex. I did not realize that there can be
multiple concurrent callers to compress_write(). One of them is the
log copying thread; further callers are data file copying threads
(default: --parallel=1).

By default, there is only one compression worker thread
(--compress-threads=1).

compress_write(): Fix a race condition between threads that would
use the same worker thread object. Make thd->data_avail contain the
thread identifier of the submitter, and add thd->avail_cond to
notify other compress_write() threads that are waiting for a slot.
2022-08-19 09:18:24 +03:00
Oleksandr Byelkin
75d631f333 Merge branch '10.7' into 10.8 2022-08-09 09:52:15 +02:00
Oleksandr Byelkin
4c18f68d59 Merge branch '10.9' into 10.10 2022-08-09 09:47:16 +02:00
Oleksandr Byelkin
50b270525a Merge branch '10.7' into 10.8 2022-08-08 17:15:13 +02:00
Oleksandr Byelkin
1d48041982 Merge branch '10.6' into 10.7 2022-08-08 17:12:32 +02:00
Oleksandr Byelkin
d2f1c3ed6c Merge branch '10.5' into bb-10.6-release 2022-08-03 12:19:59 +02:00
Oleksandr Byelkin
af143474d8 Merge branch '10.4' into 10.5 2022-08-03 07:12:27 +02:00
Oleksandr Byelkin
48e35b8cf6 Merge branch '10.3' into 10.4 2022-08-02 14:15:39 +02:00
Sergei Golubchik
5b4154373a only copy buffer pool dump in SST galera mode
and then only into the default name, so that the joiner could find it
2022-08-01 15:53:14 +02:00
Sergei Golubchik
5197519f4f revert mariabackup part of MDEV-27524, fix the test 2022-08-01 15:53:13 +02:00
Sergei Golubchik
e1caa4bd5e don't use ssl for windows named pipes - it doesn't work 2022-07-28 17:18:40 +02:00
Marko Mäkelä
4ce6e78059 Merge 10.9 into 10.10 2022-07-28 11:25:21 +03:00
Marko Mäkelä
f79cebb4d0 Merge 10.7 into 10.8 2022-07-28 10:33:26 +03:00
Marko Mäkelä
742e1c727f Merge 10.6 into 10.7 2022-07-27 18:26:21 +03:00
Marko Mäkelä
30914389fe Merge 10.5 into 10.6 2022-07-27 17:52:37 +03:00
Marko Mäkelä
098c0f2634 Merge 10.4 into 10.5 2022-07-27 17:17:24 +03:00
Oleksandr Byelkin
3bb36e9495 Merge branch '10.3' into 10.4 2022-07-27 11:02:57 +02:00
Thirunarayanan Balathandayuthapani
1d3629875e MDEV-29137 mariabackup excessive logging of ddl tracking
- Remove the FILE_MODIFY message in backup_file_op()
2022-07-26 11:33:52 +05:30
Thirunarayanan Balathandayuthapani
6156a2be30 MDEV-29137 mariabackup excessive logging of ddl tracking
- Remove the FILE_MODIFY message from mariabackup which was
displaying the list of file names which were modified since
the previous checkpoint.
2022-07-25 17:03:40 +05:30
Marko Mäkelä
b817afaa1c MDEV-28689, MDEV-28690: Remove ctrl_mutex
This reverts the revert 4f62dfe676
and fixes the hang that was introduced when ctrl_mutex was removed.

The test mariabackup.compress_qpress covers this code, but the
test is skipped if a stand-alone qpress executable is not available.
It is not available in many software repositories, possibly because
the code base has not been updated since 2010.

This was tested with an executable that was compile from the source
code at http://www.quicklz.com/qpress-11-source.zip (after adding
a missing #include <unistd.h> for the definition of isatty()).

Compared to the grandparent commit (before the revert), the changes
are as follows:

comp_thread_ctxt_t::done_cond: A separate condition for completed
compression, signaling that thd->to_len has been updated.

compress_write(): Replace some threads[i] with thd.
Reset thd->to_len = 0 after consuming the compressed data.

compress_worker_thread_func(): After consuming the uncompressed
data, set thd->data_avail = FALSE. After compressing, signal
thd->done_cond.
2022-07-11 21:00:18 +03:00
Vladislav Vaintroub
4f62dfe676 Revert "MDEV-28689, MDEV-28690: Incorrect error handling for ctrl_mutex"
This reverts commit 863c3eda87.
2022-07-11 15:00:34 +02:00
Marko Mäkelä
155019b96b MDEV-28994 Backup of memory-mapped log is corrupted
An interface to use memory-mapped I/O on the InnoDB redo log that
is stored in persistent memory was introduced
in commit 685d958e38 (MDEV-14425).

log_t::attach(): In mariadb-backup --backup, never attempt to
use memory-mapped I/O for reading the log file of the server.

xtrabackup_copy_logfile(): Assert !log_sys.is_pmem() and remove
the code to deal with a memory-mapped log.

This fixes a race condition scenario of the following type:
1. Backup parsed a mini-transaction from the memory-mapped buffer.
This took some time.
2. Meanwhile, the server might have overwritten this portion
of the circular log_sys.buf.
3. Backup copied the data to the output file while or after
the server had overwritten this portion of the file.
4. Backup failed to notice that a log overrun occurred.

The symptom of this was that a mariadb-backup --prepare of the
log failed. In the analyzed case, the error message was:
[ERROR] InnoDB: Missing FILE_CHECKPOINT(...)

This will also make it possible to run mariadb-backup --backup
under "rr replay".
2022-07-01 18:07:07 +03:00
Marko Mäkelä
d371e35257 Merge 10.9 into 10.10 2022-06-17 11:31:53 +03:00
Marko Mäkelä
cb19e211ec Merge 10.7 into 10.8 2022-06-16 11:15:21 +03:00
Marko Mäkelä
a8c22dae8b Merge 10.6 into 10.7 2022-06-16 10:50:58 +03:00
Marko Mäkelä
5bb90cb2ac Merge 10.5 into 10.6 2022-06-16 10:01:29 +03:00
Vlad Lesin
27309fc6b0 MDEV-28832 infinite loop in mariabackup if log LOG_HEADER_FORMAT field is 0
Avoid the loop with getting rid of back and forth jumping.
2022-06-15 13:30:42 +03:00
Marko Mäkelä
32edabd1f2 Merge 10.9 into 10.10 2022-06-09 15:26:09 +03:00
Marko Mäkelä
57d4a242da Merge 10.7 into 10.8 2022-06-06 16:22:09 +03:00
Marko Mäkelä
7e39470e33 Merge 10.6 into 10.7 2022-06-06 14:56:20 +03:00
Marko Mäkelä
0b47c126e3 MDEV-13542: Crashing on corrupted page is unhelpful
The approach to handling corruption that was chosen by Oracle in
commit 177d8b0c12
is not really useful. Not only did it actually fail to prevent InnoDB
from crashing, but it is making things worse by blocking attempts to
rescue data from or rebuild a partially readable table.

We will try to prevent crashes in a different way: by propagating
errors up the call stack. We will never mark the clustered index
persistently corrupted, so that data recovery may be attempted by
reading from the table, or by rebuilding the table.

This should also fix MDEV-13680 (crash on btr_page_alloc() failure);
it was extensively tested with innodb_file_per_table=0 and a
non-autoextend system tablespace.

We should now avoid crashes in many cases, such as when a page
cannot be read or allocated, or an inconsistency is detected when
attempting to update multiple pages. We will not crash on double-free,
such as on the recovery of DDL in system tablespace in case something
was corrupted.

Crashes on corrupted data are still possible. The fault injection mechanism
that is introduced in the subsequent commit may help catch more of them.

buf_page_import_corrupt_failure: Remove the fault injection, and instead
corrupt some pages using Perl code in the tests.

btr_cur_pessimistic_insert(): Always reserve extents (except for the
change buffer), in order to prevent a subsequent allocation failure.

btr_pcur_open_at_rnd_pos(): Merged to the only caller ibuf_merge_pages().

btr_assert_not_corrupted(), btr_corruption_report(): Remove.
Similar checks are already part of btr_block_get().

FSEG_MAGIC_N_BYTES: Replaces FSEG_MAGIC_N_VALUE.

dict_hdr_get(), trx_rsegf_get_new(), trx_undo_page_get(),
trx_undo_page_get_s_latched(): Replaced with error-checking calls.

trx_rseg_t::get(mtr_t*): Replaces trx_rsegf_get().

trx_rseg_header_create(): Let the caller update the TRX_SYS page if needed.

trx_sys_create_sys_pages(): Merged with trx_sysf_create().

dict_check_tablespaces_and_store_max_id(): Do not access
DICT_HDR_MAX_SPACE_ID, because it was already recovered in dict_boot().
Merge dict_check_sys_tables() with this function.

dir_pathname(): Replaces os_file_make_new_pathname().

row_undo_ins_remove_sec(): Do not modify the undo page by adding
a terminating NUL byte to the record.

btr_decryption_failed(): Report decryption failures

dict_set_corrupted_by_space(), dict_set_encrypted_by_space(),
dict_set_corrupted_index_cache_only(): Remove.

dict_set_corrupted(): Remove the constant parameter dict_locked=false.
Never flag the clustered index corrupted in SYS_INDEXES, because
that would deny further access to the table. It might be possible to
repair the table by executing ALTER TABLE or OPTIMIZE TABLE, in case
no B-tree leaf page is corrupted.

dict_table_skip_corrupt_index(), dict_table_next_uncorrupted_index(),
row_purge_skip_uncommitted_virtual_index(): Remove, and refactor
the callers to read dict_index_t::type only once.

dict_table_is_corrupted(): Remove.

dict_index_t::is_btree(): Determine if the index is a valid B-tree.

BUF_GET_NO_LATCH, BUF_EVICT_IF_IN_POOL: Remove.

UNIV_BTR_DEBUG: Remove. Any inconsistency will no longer trigger
assertion failures, but error codes being returned.

buf_corrupt_page_release(): Replaced with a direct call to
buf_pool.corrupted_evict().

fil_invalid_page_access_msg(): Never crash on an invalid read;
let the caller of buf_page_get_gen() decide.

btr_pcur_t::restore_position(): Propagate failure status to the caller
by returning CORRUPTED.

opt_search_plan_for_table(): Simplify the code.

row_purge_del_mark(), row_purge_upd_exist_or_extern_func(),
row_undo_ins_remove_sec_rec(), row_undo_mod_upd_del_sec(),
row_undo_mod_del_mark_sec(): Avoid mem_heap_create()/mem_heap_free()
when no secondary indexes exist.

row_undo_mod_upd_exist_sec(): Simplify the code.

row_upd_clust_step(), dict_load_table_one(): Return DB_TABLE_CORRUPT
if the clustered index (and therefore the table) is corrupted, similar
to what we do in row_insert_for_mysql().

fut_get_ptr(): Replace with buf_page_get_gen() calls.

buf_page_get_gen(): Return nullptr and *err=DB_CORRUPTION
if the page is marked as freed. For other modes than
BUF_GET_POSSIBLY_FREED or BUF_PEEK_IF_IN_POOL this will
trigger a debug assertion failure. For BUF_GET_POSSIBLY_FREED,
we will return nullptr for freed pages, so that the callers
can be simplified. The purge of transaction history will be
a new user of BUF_GET_POSSIBLY_FREED, to avoid crashes on
corrupted data.

buf_page_get_low(): Never crash on a corrupted page, but simply
return nullptr.

fseg_page_is_allocated(): Replaces fseg_page_is_free().

fts_drop_common_tables(): Return an error if the transaction
was rolled back.

fil_space_t::set_corrupted(): Report a tablespace as corrupted if
it was not reported already.

fil_space_t::io(): Invoke fil_space_t::set_corrupted() to report
out-of-bounds page access or other errors.

Clean up mtr_t::page_lock()

buf_page_get_low(): Validate the page identifier (to check for
recently read corrupted pages) after acquiring the page latch.

buf_page_t::read_complete(): Flag uninitialized (all-zero) pages
with DB_FAIL. Return DB_PAGE_CORRUPTED on page number mismatch.

mtr_t::defer_drop_ahi(): Renamed from mtr_defer_drop_ahi().

recv_sys_t::free_corrupted_page(): Only set_corrupt_fs()
if any log records exist for the page. We do not mind if read-ahead
produces corrupted (or all-zero) pages that were not actually needed
during recovery.

recv_recover_page(): Return whether the operation succeeded.

recv_sys_t::recover_low(): Simplify the logic. Check for recovery error.

Thanks to Matthias Leich for testing this extensively and to the
authors of https://rr-project.org for making it easy to diagnose
and fix any failures that were found during the testing.
2022-06-06 14:03:22 +03:00