Commit graph

1603 commits

Author SHA1 Message Date
Jan Lindström
f4df8c00c9 MDEV-11197: TrxInInnoDB::is_aborted(const trx_t*): Assertion `srv_read_only_mode || trx->in_depth > 0' failed
TrxInInnoDB should be constructed to track if a transaction is executing
inside InnoDB code i.e. it is like a gate between Server and InnoDB
::rnd_next() is called from Server and thus construct TrxInInnoDB
also there.

Applied suggested clean-up to TrxInInnoDB class functions enter()
and exit(). Note that exactly original did not work for enter().
2017-05-10 23:13:06 +03:00
Marko Mäkelä
021d636551 Fix some integer type mismatch.
Use uint32_t for the encryption key_id.

When filling unsigned integer values into INFORMATION_SCHEMA tables,
use the method Field::store(longlong, bool unsigned)
instead of using Field::store(double).

Fix also some miscellanous type mismatch related to ulint (size_t).
2017-05-10 12:45:46 +03:00
Marko Mäkelä
588a6a186a MDEV-12750 Fix crash recovery of key rotation
When MySQL 5.7.9 was merged to MariaDB 10.2.2, an important
debug assertion was omitted from mlog_write_initial_log_record_low().

mlog_write_initial_log_record_low(): Put back the assertion
mtr_t::is_named_space().

fil_crypt_start_encrypting_space(), fil_crypt_rotate_page():
Call mtr_t::set_named_space() before modifying any pages.

fsp_flags_try_adjust(): Call mtr_t::set_named_space(). This additional
breakage was introduced in the merge of MDEV-11623 from 10.1. It was
not caught because of the missing debug assertion in
mlog_write_initial_log_record_low().

Remove some suppressions from the encryption.innodb-redo-badkey test.
2017-05-09 21:03:27 +03:00
Daniel Black
015966d2ce MDEV-8684: innodb ut_delay - no maths
Reintroduce 3d1a7cba71
that removed the maths from ut_delay.

Existing definitions of UT_COMPILE_BARRIER used.

Signed-off-by: Daniel Black <daniel.black@au.ibm.com>
2017-05-09 13:07:15 +10:00
Marko Mäkelä
14c6f00a9f Merge 10.1 into 10.2
Also, include fixes by Vladislav Vaintroub to the
aws_key_management plugin. The AWS C++ SDK specifically depends on
OPENSSL_LIBRARIES, not generic SSL_LIBRARIES (such as YaSSL).
2017-05-06 14:36:46 +03:00
Marko Mäkelä
f9cc391863 Merge 10.1 into 10.2
This only merges MDEV-12253, adapting it to MDEV-12602 which is already
present in 10.2 but not yet in the 10.1 revision that is being merged.

TODO: Error handling in crash recovery needs to be improved.
If a page cannot be decrypted (or read), we should cleanly abort
the startup. If innodb_force_recovery is specified, we should
ignore the problematic page and apply redo log to other pages.
Currently, the test encryption.innodb-redo-badkey randomly fails
like this (the last messages are from cmake -DWITH_ASAN):

2017-05-05 10:19:40 140037071685504 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1635994
2017-05-05 10:19:40 140037071685504 [ERROR] InnoDB: Missing MLOG_FILE_NAME or MLOG_FILE_DELETE before MLOG_CHECKPOINT for tablespace 1
2017-05-05 10:19:40 140037071685504 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[2201] with error Data structure corruption
2017-05-05 10:19:41 140037071685504 [Note] InnoDB: Starting shutdown...
i=================================================================
==5226==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x612000018588 in thread T0
    #0 0x736750 in operator delete(void*) (/mariadb/server/build/sql/mysqld+0x736750)
    #1 0x1e4833f in LatchCounter::~LatchCounter() /mariadb/server/storage/innobase/include/sync0types.h:599:4
    #2 0x1e480b8 in LatchMeta<LatchCounter>::~LatchMeta() /mariadb/server/storage/innobase/include/sync0types.h:786:17
    #3 0x1e35509 in sync_latch_meta_destroy() /mariadb/server/storage/innobase/sync/sync0debug.cc:1622:3
    #4 0x1e35314 in sync_check_close() /mariadb/server/storage/innobase/sync/sync0debug.cc:1839:2
    #5 0x1dfdc18 in innodb_shutdown() /mariadb/server/storage/innobase/srv/srv0start.cc:2888:2
    #6 0x197e5e6 in innobase_init(void*) /mariadb/server/storage/innobase/handler/ha_innodb.cc:4475:3
2017-05-05 10:38:53 +03:00
Sergei Golubchik
0072d2e9a1 InnoDB cleanup: remove a bunch of #ifdef UNIV_INNOCHECKSUM
innochecksum uses global variables. great, let's use them all the
way down, instead of passing them as arguments to innodb internals,
conditionally modifying function prototypes with #ifdefs
2017-04-30 14:58:11 +02:00
Marko Mäkelä
b82c602db5 MDEV-12602 InnoDB: Failing assertion: space->n_pending_ops == 0
This fixes a regression caused by MDEV-12428.
When we introduced a variant of fil_space_acquire() that could
increment space->n_pending_ops after space->stop_new_ops was set,
the logic of fil_check_pending_operations() was broken.

fil_space_t::n_pending_ios: A new field to track read or write
access from the buffer pool routines immediately before a block
write or after a block read in the file system.

fil_space_acquire_for_io(), fil_space_release_for_io(): Similar
to fil_space_acquire_silent() and fil_space_release(), but
modify fil_space_t::n_pending_ios instead of fil_space_t::n_pending_ops.

Adjust a number of places accordingly, and remove some redundant
tablespace lookups.

The following parts of this fix differ from the 10.2 version of this fix:

buf_page_get_corrupt(): Add a tablespace parameter.

In 10.2, we already had a two-phase process of freeing fil_space objects
(first, fil_space_detach(), then release fil_system->mutex, and finally
free the fil_space and fil_node objects).

fil_space_free_and_mutex_exit(): Renamed from fil_space_free().
Detach the tablespace from the fil_system cache, release the
fil_system->mutex, and then wait for space->n_pending_ios to reach 0,
to avoid accessing freed data in a concurrent thread.
During the wait, future calls to fil_space_acquire_for_io() will
not find this tablespace, and the count can only be decremented to 0,
at which point it is safe to free the objects.

fil_node_free_part1(), fil_node_free_part2(): Refactored from
fil_node_free().
2017-04-28 14:12:52 +03:00
Marko Mäkelä
4b24467ff3 MDEV-12602 InnoDB: Failing assertion: space->n_pending_ops == 0
This fixes a regression caused by MDEV-12428.
When we introduced a variant of fil_space_acquire() that could
increment space->n_pending_ops after space->stop_new_ops was set,
the logic of fil_check_pending_operations() was broken.

fil_space_t::n_pending_ios: A new field to track read or write
access from the buffer pool routines immediately before a block
write or after a block read in the file system.

fil_space_acquire_for_io(), fil_space_release_for_io(): Similar
to fil_space_acquire_silent() and fil_space_release(), but
modify fil_space_t::n_pending_ios instead of fil_space_t::n_pending_ops.

fil_space_free_low(): Wait for space->n_pending_ios to reach 0,
to avoid accessing freed data in a concurrent thread. Future
calls to fil_space_acquire_for_io() will not find this tablespace,
because it will already have been detached from fil_system.

Adjust a number of places accordingly, and remove some redundant
tablespace lookups.

FIXME: buf_page_check_corrupt() should take a tablespace from
fil_space_acquire_for_io() as a parameter. This will be done
in the 10.1 version of this patch and merged from there.
That depends on MDEV-12253, which has not been merged from 10.1 yet.
2017-04-28 12:23:35 +03:00
Marko Mäkelä
f740d23ce6 Merge 10.1 into 10.2 2017-04-28 12:22:32 +03:00
Aditya A
14fe6dd239 Bug #23046302 COUNT(*) MUCH SLOWER ON 5.7 THAN 5.6
ANALYSIS

This is regression caused due to worklog 6742 which
implemented ha_innobase::records() which always
uses clustered index to get the row count. Previously
optimizer chose secondary index which was smaller in
size of clustered index to scan for rows and resulted in
a quicker scan.

FIX

After discussion it was decided to remove this feature in 5.7.

[#rb14040 Approved by Kevin and Oystein ]
2017-04-26 23:03:33 +03:00
Thirunarayanan Balathandayuthapani
2ef1baa75f Bug #24793413 LOG PARSING BUFFER OVERFLOW
Problem:
========
During checkpoint, we are writing all MLOG_FILE_NAME records in one mtr
and parse buffer can't be processed till MLOG_MULTI_REC_END. Eventually parse
buffer exceeds the RECV_PARSING_BUF_SIZE and eventually it overflows.

Fix:
===
1) Break the large mtr if it exceeds LOG_CHECKPOINT_FREE_PER_THREAD into multiple mtr during checkpoint.
2) Move the parsing buffer if we are encountering only MLOG_FILE_NAME
records. So that it will never exceed the RECV_PARSING_BUF_SIZE.

Reviewed-by: Debarun Bannerjee <debarun.bannerjee@oracle.com>
Reviewed-by: Rahul M Malik <rahul.m.malik@oracle.com>
RB: 14743
2017-04-26 23:03:32 +03:00
Marko Mäkelä
849af74a48 MariaDB adjustments for Oracle Bug#23070734 fix
Split the test case so that a server restart is not needed.
Reduce the test cases and use a simpler mechanism for triggering
and waiting for purge.

fil_table_accessible(): Check if a table can be accessed without
enjoying MDL protection.
2017-04-26 23:03:32 +03:00
Aditya A
62dca454e7 Bug #23070734 CONCURRENT TRUNCATE TABLES CAUSE STALLS
PROBLEM

When truncating single tablespace tables, we need to scan the entire
buffer pool to remove the pages of the table from the buffer pool.
During this scan and removal dict_sys->mutex is being held ,causing
stalls in other DDL operations.

FIX

Release the dict_sys->mutex during the scan and reacquire it after the
scan. Make sure that purge thread doesn't purge the records of the table
being truncated and background stats collection thread skips the updation
of stats for the table being truncated.

[#rb 14564 Approved by Jimmy and satya ]
2017-04-26 23:03:31 +03:00
Marko Mäkelä
13dcdb0903 MDEV-11802 InnoDB purge does not always run when there is work to do
srv_sys_t::n_threads_active[]: Protect writes by both the mutex and
by atomic memory access.

srv_active_wake_master_thread_low(): Reliably wake up the master
thread if there is work to do. The trick is to atomically read
srv_sys->n_threads_active[].

srv_wake_purge_thread_if_not_active(): Atomically read
srv_sys->n_threads_active[] (and trx_sys->rseg_history_len),
so that the purge should always be triggered when there is work to do.

trx_commit_in_memory(): Invoke srv_wake_purge_thread_if_not_active()
whenever a transaction is committed. Purge could have been prevented by
the read view of the currently committing transaction, even if it is
a read-only transaction.

trx_purge_add_update_undo_to_history(): Do not wake up the purge.
This is only called by trx_undo_update_cleanup(), as part of
trx_write_serialisation_history(), which in turn is only called by
trx_commit_low() which will always call trx_commit_in_memory().
Thus, the added call in trx_commit_in_memory() will cover also
this use case where a committing read-write transaction added
some update_undo log to the purge queue.

trx_rseg_mem_restore(): Atomically modify trx_sys->rseg_history_len.
2017-04-26 23:03:31 +03:00
Marko Mäkelä
6264e892cf Remove redundant initialization of global InnoDB variables
Also, remove the unused global variable srv_priority_boost.
2017-04-26 23:03:31 +03:00
Marko Mäkelä
797db28d53 Remove the unused declaration fsp_is_temporary() 2017-04-26 23:03:31 +03:00
Darshan M N
698e37d60c BUG#25251082 DISABLING CERTAIN MACROS IN INNODB RESULTS IN COMPILATION ERRORS
Issue:
======
Disabling macros such as UNIV_PFS_MUTEX/UNIV_PFS_RWLOCK/UNIV_PFS_THREAD
which are defined in InnoDB throws errors during compilation.

Fix:
====
Fix all the compilation errors.

RB: 14893
Reviewed-by: Jimmy Yang <Jimmy.Yang@oracle.com>
Reviewed-by: Satya Bodapati <satya.bodapati@oracle.com>
2017-04-26 23:03:30 +03:00
Debarun Banerjee
bf5be32376 BUG#25032066 PREPARED TRANSACTION SHOULD NOT BE ROLLED BACK BY HIGH PRIORITY TRANSACTION
Problem :
---------
1. delete_all_rows() and rnd_init() are not returning error
after async rollback in 5.7. This results in assert in
innodb in next call.

2. High priority transaction is rolling back prepared transaction.
This is because TRX_FORCE_ROLLBACK_DISABLE is getting set only for
first entry [TrxInInnoDB].

Solution :
----------
1. return DB_FORCED_ABORT error after rollback.
2. check and disable rollback in TrxInInnodb::enter always.

Reviewed-by: Sunny Bains <sunny.bains@oracle.com>

RB: 13777
2017-04-26 23:03:29 +03:00
Vasil Dimov
6c76e5a00b Fix Bug#24605956 SERVER MAY CRASH DUE TO A GLIBC BUG IN HANDLING SHORT-LIVED DETACHED THREADS
Avoid detaching and exiting from threads that may finish before the
caller has returned from pthread_create(). Only exit from such threads,
without detach and join with them later.

Patch submitted by: Laurynas Biveinis <laurynas.biveinis@gmail.com>
RB: 13983
Reviewed by: Sunny Bains <sunny.bains@oracle.com>
2017-04-26 23:03:29 +03:00
Marko Mäkelä
da76c1bd3e Minor cleanup 2017-04-26 23:03:28 +03:00
Marko Mäkelä
206ecb79a5 Follow-up to MDEV-12289: Support innodb_undo_tablespaces=127
MySQL 5.7 reduced the maximum number of innodb_undo_tablespaces
from 126 to 95 when it reserved 32 persistent rollback segments
for the temporary undo logs. Since MDEV-12289 restored all 128
persistent rollback segments for persistent undo logs, the
reasonable maximum value of innodb_undo_tablespaces is 127
(not 126 or 95). This is because out of the 128 rollback segments,
the first one will always be created in the system tablespace
and the remaining ones can be created in dedicated undo tablespaces.
2017-04-26 23:03:28 +03:00
Marko Mäkelä
23ea4360fd Remove TRX_SYS_OLD_N_RSEGS 2017-04-26 23:03:27 +03:00
Marko Mäkelä
472b5f0d1f Follow-up to Bug#24346574 PAGE CLEANER THREAD, ASSERT BLOCK->N_POINTERS == 0
Silence the Valgrind warnings on instrumented builds (-DWITH_VALGRIND).

assert_block_ahi_empty_on_init(): A variant of
assert_block_ahi_empty() that declares n_pointers initialized and then
asserts that n_pointers==0.

In Valgrind-instrumented builds, InnoDB declares allocated memory
uninitialized.
2017-04-26 23:03:27 +03:00
Marko Mäkelä
e63ead68bf Bug#24346574 PAGE CLEANER THREAD, ASSERT BLOCK->N_POINTERS == 0
btr_search_drop_page_hash_index(): Do not return before ensuring
that block->index=NULL, even if !btr_search_enabled. We would
typically still skip acquiring the AHI latch when the AHI is
disabled, because block->index would already be NULL. Only if the AHI
is in the process of being disabled, we would wait for the AHI latch
and then notice that block->index=NULL and return.

The above bug was a regression caused in MySQL 5.7.9 by the fix of
Bug#21407023: DISABLING AHI SHOULD AVOID TAKING AHI LATCH

The rest of this patch improves diagnostics by adding assertions.

assert_block_ahi_valid(): A debug predicate for checking that
block->n_pointers!=0 implies block->index!=NULL.

assert_block_ahi_empty(): A debug predicate for checking that
block->n_pointers==0.

buf_block_init(): Instead of assigning block->n_pointers=0,
assert_block_ahi_empty(block).

buf_pool_clear_hash_index(): Clarify comments, and assign
block->n_pointers=0 before assigning block->index=NULL.
The wrong ordering could make block->n_pointers appear incorrect in
debug assertions. This bug was introduced in MySQL 5.1.52 by
Bug#13006367 62487: INNODB TAKES 3 MINUTES TO CLEAN UP THE
ADAPTIVE HASH INDEX AT SHUTDOWN

i_s_innodb_buffer_page_get_info(): Add a comment that
the IS_HASHED column in the INFORMATION_SCHEMA views
INNODB_BUFFER_POOL_PAGE and INNODB_BUFFER_PAGE_LRU may
show false positives (there may be no pointers after all.)

ha_insert_for_fold_func(), ha_delete_hash_node(),
ha_search_and_update_if_found_func(): Use atomics for
updating buf_block_t::n_pointers. While buf_block_t::index is
always protected by btr_search_x_lock(index), in
ha_insert_for_fold_func() the n_pointers-- may belong to
another dict_index_t whose btr_search_latches[] we are not holding.

RB: 13879
Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>
2017-04-26 23:03:27 +03:00
Jan Lindström
765a43605a MDEV-12253: Buffer pool blocks are accessed after they have been freed
Problem was that bpage was referenced after it was already freed
from LRU. Fixed by adding a new variable encrypted that is
passed down to buf_page_check_corrupt() and used in
buf_page_get_gen() to stop processing page read.

This patch should also address following test failures and
bugs:

MDEV-12419: IMPORT should not look up tablespace in
PageConverter::validate(). This is now removed.

MDEV-10099: encryption.innodb_onlinealter_encryption fails
sporadically in buildbot

MDEV-11420: encryption.innodb_encryption-page-compression
failed in buildbot

MDEV-11222: encryption.encrypt_and_grep failed in buildbot on P8

Removed dict_table_t::is_encrypted and dict_table_t::ibd_file_missing
and replaced these with dict_table_t::file_unreadable. Table
ibd file is missing if fil_get_space(space_id) returns NULL
and encrypted if not. Removed dict_table_t::is_corrupted field.

Ported FilSpace class from 10.2 and using that on buf_page_check_corrupt(),
buf_page_decrypt_after_read(), buf_page_encrypt_before_write(),
buf_dblwr_process(), buf_read_page(), dict_stats_save_defrag_stats().

Added test cases when enrypted page could be read while doing
redo log crash recovery. Also added test case for row compressed
blobs.

btr_cur_open_at_index_side_func(),
btr_cur_open_at_rnd_pos_func(): Avoid referencing block that is
NULL.

buf_page_get_zip(): Issue error if page read fails.

buf_page_get_gen(): Use dberr_t for error detection and
do not reference bpage after we hare freed it.

buf_mark_space_corrupt(): remove bpage from LRU also when
it is encrypted.

buf_page_check_corrupt(): @return DB_SUCCESS if page has
been read and is not corrupted,
DB_PAGE_CORRUPTED if page based on checksum check is corrupted,
DB_DECRYPTION_FAILED if page post encryption checksum matches but
after decryption normal page checksum does not match. In read
case only DB_SUCCESS is possible.

buf_page_io_complete(): use dberr_t for error handling.

buf_flush_write_block_low(),
buf_read_ahead_random(),
buf_read_page_async(),
buf_read_ahead_linear(),
buf_read_ibuf_merge_pages(),
buf_read_recv_pages(),
fil_aio_wait():
        Issue error if page read fails.

btr_pcur_move_to_next_page(): Do not reference page if it is
NULL.

Introduced dict_table_t::is_readable() and dict_index_t::is_readable()
that will return true if tablespace exists and pages read from
tablespace are not corrupted or page decryption failed.
Removed buf_page_t::key_version. After page decryption the
key version is not removed from page frame. For unencrypted
pages, old key_version is removed at buf_page_encrypt_before_write()

dict_stats_update_transient_for_index(),
dict_stats_update_transient()
        Do not continue if table decryption failed or table
        is corrupted.

dict0stats.cc: Introduced a dict_stats_report_error function
to avoid code duplication.

fil_parse_write_crypt_data():
        Check that key read from redo log entry is found from
        encryption plugin and if it is not, refuse to start.

PageConverter::validate(): Removed access to fil_space_t as
tablespace is not available during import.

Fixed error code on innodb.innodb test.

Merged test cased innodb-bad-key-change5 and innodb-bad-key-shutdown
to innodb-bad-key-change2.  Removed innodb-bad-key-change5 test.
Decreased unnecessary complexity on some long lasting tests.

Removed fil_inc_pending_ops(), fil_decr_pending_ops(),
fil_get_first_space(), fil_get_next_space(),
fil_get_first_space_safe(), fil_get_next_space_safe()
functions.

fil_space_verify_crypt_checksum(): Fixed bug found using ASAN
where FIL_PAGE_END_LSN_OLD_CHECKSUM field was incorrectly
accessed from row compressed tables. Fixed out of page frame
bug for row compressed tables in
fil_space_verify_crypt_checksum() found using ASAN. Incorrect
function was called for compressed table.

Added new tests for discard, rename table and drop (we should allow them
even when page decryption fails). Alter table rename is not allowed.
Added test for restart with innodb-force-recovery=1 when page read on
redo-recovery cant be decrypted. Added test for corrupted table where
both page data and FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION is corrupted.

Adjusted the test case innodb_bug14147491 so that it does not anymore
expect crash. Instead table is just mostly not usable.

fil0fil.h: fil_space_acquire_low is not visible function
and fil_space_acquire and fil_space_acquire_silent are
inline functions. FilSpace class uses fil_space_acquire_low
directly.

recv_apply_hashed_log_recs() does not return anything.
2017-04-26 15:19:16 +03:00
Aditya A
44b1fb3614 WL9513 Bug#23333990 PERSISTENT INDEX STATISTICS UPDATE BEFORE TRANSACTION IS COMMITTED
PROBLEM

By design stats estimation always reading uncommitted data. In this scenario
an uncommitted transaction has deleted all rows in the table. In Innodb
uncommitted delete records are marked as delete but not actually removed
from Btree until the transaction has committed or a read view for the rows
is present.While calculating persistent stats we were ignoring the delete
marked records,since all the records are delete marked we were estimating
the number of rows present in the table as zero which leads to bad plans
in other transaction operating on the table.

Fix

Introduced a system variable called innodb_stats_include_delete_marked
which when enabled includes delete marked records for stat
calculations .
2017-04-24 17:45:23 +03:00
Shaohua Wang
d3a2f60e1a BUG#23477773 OPTION TO TURN OFF/ON DEADLOCK CHECKER
Backport WL#9383 INNODB: ADD AN OPTION TO TURN OFF/ON DEADLOCK CHECKER
(rb#12873) to 5.7.
2017-04-24 15:09:18 +03:00
Shaohua Wang
1a5ca702b1 Followup: BUG#23479595 SEGMENTATION FAULT WHEN SELECT FTS INDEX TABLES IN INFORMATION SCHEMA
BUG#23742339 FAILING ASSERTION: SYM_NODE->TABLE != NULL

Analysis: When we access fts aux tables in information schema,the
fts aux tables are dropped by DROP DATABASE in another session.

Solution: Drop parent table if it's a fts aux table, and drop
table will drop fts aux tables together.

Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>
RB: 13264
2017-04-24 14:53:59 +03:00
Shaohua Wang
9ce1ea6f51 BUG#24009272 SEGFAULT WITH CREATE+SELECT FROM IS+DROP FTS TABLE CONCURRENTLY
Analysis:
When we access fts_internal_tbl_name in i_s_fts_config_fill (),
it can be set to NULL by another session.

Solution:
Define fts_internal_tbl_name2 for global variable innodb_ft_aux_table,
if it's NULL, set fts_internal_tbl_name to "default".

Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>
RB: 13401
2017-04-24 14:53:22 +03:00
Marko Mäkelä
200ef51344 Fix a compilation error 2017-04-21 18:29:50 +03:00
Marko Mäkelä
11f772fad6 Remove an orphan declaration of os_get_os_version() 2017-04-21 18:12:20 +03:00
Marko Mäkelä
0871a00a62 MDEV-12545 Reduce the amount of fil_space_t lookups
buf_flush_write_block_low(): Acquire the tablespace reference once,
and pass it to lower-level functions. This is only a start; further
calls may be removed.

fil_decompress_page(): Remove unsafe use of fil_space_get_by_id().
2017-04-21 18:12:10 +03:00
Marko Mäkelä
5684aa220c MDEV-12488 Remove type mismatch in InnoDB printf-like calls
Alias the InnoDB ulint and lint data types to size_t and ssize_t,
which are the standard names for the machine-word-width data types.

Correspondingly, define ULINTPF as "%zu" and introduce ULINTPFx as "%zx".
In this way, better compiler warnings for type mismatch are possible.

Furthermore, use PRIu64 for that 64-bit format, and define
the feature macro __STDC_FORMAT_MACROS to enable it on Red Hat systems.

Fix some errors in error messages, and replace some error messages
with assertions.
Most notably, an IMPORT TABLESPACE error message in InnoDB was
displaying the number of columns instead of the mismatching flags.
2017-04-21 18:03:15 +03:00
Marko Mäkelä
039a299b92 MDEV-12534 Use atomic operations whenever available
Define UNIV_WORD_SIZE as a simple alias to SIZEOF_SIZE_T.
In MariaDB 10.0 and 10.1, it was incorrectly defined as 4 on
64-bit Windows.

MONITOR_OS_PENDING_READS, MONITOR_OS_PENDING_WRITES: Enable by default.

os_n_pending_reads, os_n_pending_writes: Remove.
Use the monitor counters instead.
2017-04-21 17:55:22 +03:00
Marko Mäkelä
996c7d5cb5 MDEV-12545 Reduce the amount of fil_space_t lookups
buf_flush_write_block_low(): Acquire the tablespace reference once,
and pass it to lower-level functions. This is only a start; further
calls may be removed later.
2017-04-21 17:47:23 +03:00
Marko Mäkelä
d23eb8e6d2 Follow-up to MDEV-12488: Fix some type mismatch in header files
This reduces the number of compilation warnings on Windows.
2017-04-21 17:47:05 +03:00
Marko Mäkelä
8c38147cdd Merge 10.0 into 10.1 2017-04-21 12:46:12 +03:00
Marko Mäkelä
87b6df31c4 MDEV-12488 Remove type mismatch in InnoDB printf-like calls
This is a reduced version of an originally much larger patch.
We will keep the definition of the ulint, lint data types unchanged,
and we will not be replacing fprintf() calls with ib_logf().

On Windows, use the standard format strings instead of nonstandard
extensions.

This patch fixes some errors in format strings.
Most notably, an IMPORT TABLESPACE error message in InnoDB was
displaying the number of columns instead of the mismatching flags.
2017-04-21 12:06:29 +03:00
Marko Mäkelä
d34a67b067 MDEV-12534 Use atomic operations whenever available
Allow 64-bit atomic operations on 32-bit systems,
only relying on HAVE_ATOMIC_BUILTINS_64, disregarding
the width of the register file.

Define UNIV_WORD_SIZE correctly on all systems, including Windows.
In MariaDB 10.0 and 10.1, it was incorrectly defined as 4 on
64-bit Windows.

Define HAVE_ATOMIC_BUILTINS_64 on Windows
(64-bit atomics are available on both 32-bit and 64-bit Windows
platforms; the operations were unnecessarily disabled even on
64-bit Windows).

MONITOR_OS_PENDING_READS, MONITOR_OS_PENDING_WRITES: Enable by default.

os_file_n_pending_preads, os_file_n_pending_pwrites,
os_n_pending_reads, os_n_pending_writes: Remove.
Use the monitor counters instead.

os_file_count_mutex: Remove. On a system that does not support
64-bit atomics, monitor_mutex will be used instead.
2017-04-20 16:29:12 +03:00
Marko Mäkelä
3a50fd9d2f Simplify page_is_comp(). 2017-04-17 03:20:38 +03:00
Marko Mäkelä
a039d6c3e6 Remove unused tokens from the InnoDB SQL parser.
PARS_INTEGER_TOKEN: Remove. The lexer returns only PARS_INT_TOKEN.

PARS_FIXBINARY_LIT, PARS_BLOB_LIT: Remove. These are never returned
by the lexer. In sym_tab_add_bound_lit(), use PARS_STR_LIT.
2017-04-17 03:20:34 +03:00
Marko Mäkelä
e5eef05140 Minor code cleanup.
dict_index_is_sec_or_ibuf(): Use a single arithmetic expression.

rtr_split_page_move_rec_list(): Remove a redundant condition on
dict_index_is_sec_or_ibuf(). This function is always invoked on
a spatial index, which also is a secondary index.
2017-04-17 03:20:34 +03:00
Marko Mäkelä
85da56bf2d Remove the unused variable trx_t::support_xa.
Also, merge some test changes from MySQL 5.7.10, for deprecating
innodb_support_xa.
2017-04-07 13:33:59 +03:00
Sachin Setiya
cdd1dc829b MW-28, codership/mysql-wsrep#28 Fix sync_thread_levels debug assert
Introduced a new wsrep_trx_print_locking() which may be called
under lock_sys->mutex if the trx has locks.

Signed-off-by: Sachin Setiya <sachin.setiya@mariadb.com>
2017-04-06 15:41:54 +05:30
Marko Mäkelä
6e105d7535 Merge 10.1 into 10.2 2017-04-04 07:59:25 +03:00
Marko Mäkelä
9505c96839 MDEV-12428 SIGSEGV in buf_page_decrypt_after_read() during DDL
Also, some MDEV-11738/MDEV-11581 post-push fixes.

In MariaDB 10.1, there is no fil_space_t::is_being_truncated field,
and the predicates fil_space_t::stop_new_ops and fil_space_t::is_stopping()
are interchangeable. I requested the fil_space_t::is_stopping() to be added
in the review, but some added checks for fil_space_t::stop_new_ops were
not replaced with calls to fil_space_t::is_stopping().

buf_page_decrypt_after_read(): In this low-level I/O operation, we must
look up the tablespace if it exists, even though future I/O operations
have been blocked on it due to a pending DDL operation, such as DROP TABLE
or TRUNCATE TABLE or other table-rebuilding operations (ALTER, OPTIMIZE).
Pass a parameter to fil_space_acquire_low() telling that we are performing
a low-level I/O operation and the fil_space_t::is_stopping() status should
be ignored.
2017-04-03 22:09:28 +03:00
Marko Mäkelä
124bae082b MDEV-12289 Keep 128 persistent rollback segments for compatibility and performance
InnoDB divides the allocation of undo logs into rollback segments.
The DB_ROLL_PTR system column of clustered indexes can address up to
128 rollback segments (TRX_SYS_N_RSEGS). Originally, InnoDB only
created one rollback segment. In MySQL 5.5 or in the InnoDB Plugin
for MySQL 5.1, all 128 rollback segments were created.

MySQL 5.7 hard-codes the rollback segment IDs 1..32 for temporary undo logs.
On upgrade, unless a slow shutdown (innodb_fast_shutdown=0)
was performed on the old server instance, these rollback segments
could be in use by transactions that are in XA PREPARE state or
transactions that were left behind by a server kill followed by a
normal shutdown immediately after restart.

Persistent tables cannot refer to temporary undo logs or vice versa.
Therefore, we should keep two distinct sets of rollback segments:
one for persistent tables and another for temporary tables. In this way,
all 128 rollback segments will be available for both types of tables,
which could improve performance. Also, MariaDB 10.2 will remain more
compatible than MySQL 5.7 with data files from earlier versions of
MySQL or MariaDB.

trx_sys_t::temp_rsegs[TRX_SYS_N_RSEGS]: A new array of temporary
rollback segments. The trx_sys_t::rseg_array[TRX_SYS_N_RSEGS] will
be solely for persistent undo logs.

srv_tmp_undo_logs. Remove. Use the constant TRX_SYS_N_RSEGS.

srv_available_undo_logs: Change the type to ulong.

trx_rseg_get_on_id(): Remove. Instead, let the callers refer to
trx_sys directly.

trx_rseg_create(), trx_sysf_rseg_find_free(): Remove unneeded parameters.
These functions only deal with persistent undo logs.

trx_temp_rseg_create(): New function, to create all temporary rollback
segments at server startup.

trx_rseg_t::is_persistent(): Determine if the rollback segment is for
persistent tables.

trx_sys_is_noredo_rseg_slot(): Remove. The callers must know based on
context (such as table handle) whether the DB_ROLL_PTR is referring to
a persistent undo log.

trx_sys_create_rsegs(): Remove all parameters, which were always passed
as global variables. Instead, modify the global variables directly.

enum trx_rseg_type_t: Remove.

trx_t::get_temp_rseg(): A method to ensure that a temporary
rollback segment has been assigned for the transaction.

trx_t::assign_temp_rseg(): Replaces trx_assign_rseg().

trx_purge_free_segment(), trx_purge_truncate_rseg_history():
Remove the redundant variable noredo=false.
Temporary undo logs are discarded immediately at transaction commit
or rollback, not lazily by purge.

trx_purge_mark_undo_for_truncate(): Remove references to the
temporary rollback segments.

trx_purge_mark_undo_for_truncate(): Remove a check for temporary
rollback segments. Only the dedicated persistent undo log tablespaces
can be truncated.

trx_undo_get_undo_rec_low(), trx_undo_get_undo_rec(): Add the
parameter is_temp.

trx_rseg_mem_restore(): Split from trx_rseg_mem_create().
Initialize the undo log and the rollback segment from the file
data structures.

trx_sysf_get_n_rseg_slots(): Renamed from
trx_sysf_used_slots_for_redo_rseg(). Count the persistent
rollback segment headers that have been initialized.

trx_sys_close(): Also free trx_sys->temp_rsegs[].

get_next_redo_rseg(): Merged to trx_assign_rseg_low().

trx_assign_rseg_low(): Remove the parameters and access the
global variables directly. Revert to simple round-robin, now that
the whole trx_sys->rseg_array[] is for persistent undo log again.

get_next_noredo_rseg(): Moved to trx_t::assign_temp_rseg().

srv_undo_tablespaces_init(): Remove some parameters and use the
global variables directly. Clarify some error messages.

Adjust the test innodb.log_file. Apparently, before these changes,
InnoDB somehow ignored missing dedicated undo tablespace files that
are pointed by the TRX_SYS header page, possibly losing part of
essential transaction system state.
2017-03-31 18:53:04 +03:00
Marko Mäkelä
a0c79bcf00 Rename InnoDB transaction undo logging predicates.
trx::has_logged_persistent(): Renamed from trx_is_redo_rseg_updated().
Determines if a transaction has generated any persistent undo log.

trx::has_logged(): Renamed from trx_is_rseg_updated().
Determines if a transaction has generated any undo log.
2017-03-30 14:35:19 +03:00
Sergei Golubchik
da4d71d10d Merge branch '10.1' into 10.2 2017-03-30 12:48:42 +02:00