Commit graph

1466 commits

Author SHA1 Message Date
Vladislav Vaintroub
1ca8637ae3 Windows : Fix several thousand of warnings with Visual C++ compiler
On some weird reason, Visual C++ does not like newly introduced
#define ut_ad(EXPR)	DBUG_ASSERT(EXPR)

and writes bogus "not enough parameters for DBUG_ASSERT" warning.

Workaround it with
#define ut_ad DBUG_ASSERT
2017-03-20 14:14:52 +00:00
Marko Mäkelä
97acc4a1c3 MDEV-12270 Port MySQL 8.0 Bug#21141390 REMOVE UNUSED FUNCTIONS AND CONVERT GLOBAL SYMBOLS TO STATIC
InnoDB defines some functions that are not called at all.
Other functions are called, but only from the same compilation unit.

Remove some function declarations and definitions, and add 'static'
keywords. Some symbols must be kept for separately compiled tools,
such as innochecksum.
2017-03-17 12:48:50 +02:00
Marko Mäkelä
4e1116b2c6 MDEV-12271 Port MySQL 8.0 Bug#23150562 REMOVE UNIV_MUST_NOT_INLINE AND UNIV_NONINL
Also, remove empty .ic files that were not removed by my MySQL commit.

Problem:
InnoDB used to support a compilation mode that allowed to choose
whether the function definitions in .ic files are to be inlined or not.
This stopped making sense when InnoDB moved to C++ in MySQL 5.6
(and ha_innodb.cc started to #include .ic files), and more so in
MySQL 5.7 when inline methods and functions were introduced
in .h files.

Solution:
Remove all references to UNIV_NONINL and UNIV_MUST_NOT_INLINE from
all files, assuming that the symbols are never defined.
Remove the files fut0fut.cc and ut0byte.cc which only mattered when
UNIV_NONINL was defined.
2017-03-17 12:42:07 +02:00
Marko Mäkelä
7668a79a88 MDEV-12269 Port Bug#22996442 INNODB: MAKE UNIV_DEBUG DEPEND ON DBUG_OFF
This is a partial port of my patch in MySQL 8.0.
In MySQL 8.0, all InnoDB references to DBUG_OFF were replaced
with UNIV_DEBUG. We will not do that in MariaDB.

InnoDB used two independent compile-time flags that distinguish
debug and non-debug builds, which is confusing.

Also, make ut_ad() and alias of DBUG_ASSERT().
2017-03-16 10:24:53 +02:00
Marko Mäkelä
105f46ffb8 MDEV-12273 Remove dict_table_t::does_not_fit_in_memory
In the InnoDB internal SQL parser, there is the keyword
DOES_NOT_FIT_IN_MEMORY that is never specified in any CREATE TABLE
statement that is passed to the InnoDB SQL parser
(que_eval_sql() or pars_sql() or yyparse()). If this keyword were
ever present, it would set the flag dict_table_t::does_not_fit_in_memory
which is only present in debug builds.

Let us remove all traces of this.

Also, fix storage/innobase/pars/make_flex.sh so that no the generated
file storage/innobase/pars/lexyy.cc works as is.

FIXME: Always generate the InnoDB Bison files at build time, similar
to how sql/sql_yacc.yy is handled. (This would still leave the
generated scanner files, unless we want to add a build-time dependency
for Flex.)
2017-03-16 10:24:53 +02:00
Marko Mäkelä
13e5c9de80 MDEV-12219 Discard temporary undo logs at transaction commit
Starting with MySQL 5.7, temporary tables in InnoDB are handled
differently from persistent tables. Because temporary tables are
private to a connection, concurrency control and multi-versioning
(MVCC) are not applicable. For performance reasons, purge is
disabled as well. Rollback is supported for temporary tables;
that is why we have the temporary undo logs in the first place.

Because MVCC and purge are disabled for temporary tables, we should
discard all temporary undo logs already at transaction commit,
just like we discard the persistent insert_undo logs. Before this
change, update_undo logs were being preserved.

trx_temp_undo_t: A wrapper for temporary undo logs, comprising
a rollback segment and a single temporary undo log.

trx_rsegs_t::m_noredo: Use trx_temp_undo_t.
(Instead of insert_undo, update_undo, there will be a single undo.)

trx_is_noredo_rseg_updated(), trx_is_rseg_assigned(): Remove.

trx_undo_add_page(): Remove the parameter undo_ptr.
Acquire and release the rollback segment mutex inside the function.

trx_undo_free_last_page(): Remove the parameter trx.

trx_undo_truncate_end(): Remove the parameter trx, and add the
parameter is_temp. Clean up the code a bit.

trx_undo_assign_undo(): Split the parameter undo_ptr into rseg, undo.

trx_undo_commit_cleanup(): Renamed from trx_undo_insert_cleanup().
Replace the parameter undo_ptr with undo.
This will discard the temporary undo or insert_undo log at
commit/rollback.

trx_purge_add_update_undo_to_history(), trx_undo_update_cleanup():
Remove 3 parameters. Always operate on the persistent update_undo.

trx_serialise(): Renamed from trx_serialisation_number_get().

trx_write_serialisation_history(): Simplify the code flow.
If there are no persistent changes, do not update MONITOR_TRX_COMMIT_UNDO.

trx_commit_in_memory(): Simplify the logic, and add assertions.

trx_undo_page_report_modify(): Keep a direct reference to the
persistent update_undo log.

trx_undo_report_row_operation(): Simplify some code.
Always assign TRX_UNDO_INSERT for temporary undo logs.

trx_prepare_low(): Keep only one parameter. Prepare all 3 undo logs.

trx_roll_try_truncate(): Remove the parameter undo_ptr.
Try to truncate all 3 undo logs of the transaction.

trx_roll_pop_top_rec_of_trx_low(): Remove.

trx_roll_pop_top_rec_of_trx(): Remove the redundant parameter
trx->roll_limit. Clear roll_limit when exhausting the undo logs.
Consider all 3 undo logs at once, prioritizing the persistent
undo logs.

row_undo(): Minor cleanup. Let trx_roll_pop_top_rec_of_trx()
reset the trx->roll_limit.
2017-03-13 18:57:17 +02:00
Marko Mäkelä
056ec4ab24 Fix some compilation warnings. 2017-03-13 18:11:36 +02:00
Marko Mäkelä
ff8bf6e933 Define a mtr_t::start() wrapper inline. 2017-03-13 18:11:01 +02:00
Marko Mäkelä
c32dcae65a Adjust an outdated comment.
os_sync_free(), which hid resource leaks, was removed in MySQL 5.7.
2017-03-13 18:11:01 +02:00
Marko Mäkelä
a20340cf85 Hard-code innodb_page_size as the undo log page size.
InnoDB undo logs currently always use the innodb_page_size,
whether they are stored in the system tablespace, in a
dedicated undo tablespace, or in the temporary tablespace.
Remove redundant page_size parameters.

TrxUndoRsegsIterator::set_next(): return bool instead of page_size.
2017-03-10 08:15:25 +02:00
Marko Mäkelä
1417839810 InnoDB purge_sys cleanup.
TrxUndoRsegsIterator::m_purge_sys: Remove. There is only one purge_sys.

purge_sys_t: Renamed from trx_purge_t. Define a constructor and
destructor. Allocate rseg_iter, purge_queue inline.

purge_sys->trx: Remove. Use purge_sys->sess->trx instead.

purge_sys->view_active: Remove. Access to purge_sys->view is always
protected by purge_sys->latch.

trx_purge_sys_create(): Replaced by purge_sys_t::purge_sys_t().

trx_purge_sys_close(): Replaced by purge_sys_t::~purge_sys_t().
2017-03-10 08:15:25 +02:00
Vladislav Vaintroub
a98009ab02 MDEV-12201 innodb_flush_method are not available on Windows
Remove srv_win_file_flush_method

- Rename srv_unix_file_flush_method to srv_file_flush_method, and
  rename constants to remove UNIX from them, i.e SRV_UNIX_FSYNC=>SRV_FSYNC

- Add SRV_ALL_O_DIRECT_FSYNC corresponding to current Windows default
(no buffering for either log or data, flush on both log and data)

- change os_file_open on Windows to behave identically to Unix wrt
O_DIRECT and O_DSYNC settings. map O_DIRECT to FILE_FLAG_NO_BUFFERING and
O_DSYNC to FILE_FLAG_WRITE_THROUGH

- remove various #ifdef _WIN32
2017-03-09 19:19:38 +00:00
Marko Mäkelä
70a0500d3c Remove some InnoDB purge definitions from trx0types.h. 2017-03-09 17:35:09 +02:00
Marko Mäkelä
7a30d86e9d Simplify InnoDB startup.
InnoDB needs to collect transactions from the persistent data files
in trx_rseg_array_init() before trx_lists_init_at_db_start() is
executed. But there is no need to create purge_sys->purge_queue
separately from purge_sys.

trx_sys_init_at_db_start(): Change the return type to void.
Remove the direct call to trx_rseg_array_init(). It will be called
by trx_lists_init_at_db_start(), which we are calling.
Initialize the purge system read view.

trx_lists_init_at_db_start(): Call trx_purge_sys_create(), which will
invoke trx_rseg_array_init() to read the undo log segments.

trx_purge_sys_create(): Remove the parameters. Do not initialize
the purge system read view, because trx_sys->rw_trx_list has not
been recovered yet. The purge_sys->view will be initialized at
the end of trx_sys_init_at_db_start().

trx_rseg_array_init(): Remove the parameter. Use purge_sys->purge_queue
directly.

innobase_start_or_create_for_mysql(): Remove the local variable
purge_queue. Do not call trx_purge_sys_create(), because it will be
called by trx_sys_init_at_db_start().
2017-03-09 17:28:06 +02:00
Marko Mäkelä
15bdfeeba8 Remove trx_sys_t::pending_purge_rseg_array.
In MySQL 5.7, there is some redundant code for supposedly handling
an upgrade from an earlier version of InnoDB.

An upgrade of InnoDB between major versions should include a
slow shutdown (innodb_fast_shutdown=0) of the previous version.

A comment in trx_lists_init_at_db_start() confused clean shutdown
and slow shutdown. A clean shutdown does not necessarily guarantee
that there are no active transactions. A slow shutdown guarantees
that.

Because there was no code to handle rollback of recovered transactions
that happened to use the rollback segment slots that MySQL 5.7.2
repurposed for temporary undo logs, the upgrade is not working in all
cases, and we may as well remove the code to handle purging.

trx_sys_t::pending_purge_rseg_array: Remove.

trx_undo_get_undo_rec_low(): Define as static. Remove the parameter
is_redo_rseg.

trx_undo_get_undo_rec(), trx_rseg_get_on_id(): Remove the parameter
is_redo_rseg.

trx_rseg_mem_free(): Remove the second parameter.

trx_sys_get_nth_rseg(): Replace with trx_rseg_get_on_id().

trx_rseg_schedule_pending_purge(): Remove.
2017-03-09 15:58:33 +02:00
Marko Mäkelä
5ff6694d70 enum btr_latch_mode: Incorporate some flags.
This fixes some GCC 6.3.0 warnings and makes the code a little
more debugging-friendly.
2017-03-09 10:30:36 +02:00
Marko Mäkelä
5da6bd7b95 MDEV-11027 InnoDB log recovery is too noisy
Provide more useful progress reporting of crash recovery.

recv_sys_t::progress_time: The time of the last report.

recv_sys_t::report(ib_time_t): Determine whether progress should
be reported.

recv_scan_print_counter: Remove.

log_group_read_log_seg(): After after each I/O request, invoke
recv_sys_t::report() and report progress if needed.

recv_apply_hashed_log_recs(): Change the return type back to void
(DB_SUCCESS was always returned), and rename the parameter to last_batch.
At the start of each batch, if there are pages to be recovered,
issue a message.
2017-03-08 14:55:11 +02:00
Marko Mäkelä
2bca41265c Remove MLOG_COMP_REC_SEC_DELETE_MARK.
The InnoDB redo log record type MLOG_COMP_REC_SEC_DELETE_MARK has
been unused for a long time, and it has never been written after
WL#8845 introduced the redo log format identifier in MySQL 5.7.9
or MariaDB Server 10.2.2. Thus, removing the record type does not
constitute any functional change.
2017-03-07 19:07:41 +02:00
Marko Mäkelä
89d80c1b0b Fix many -Wconversion warnings.
Define my_thread_id as an unsigned type, to avoid mismatch with
ulonglong.  Change some parameters to this type.

Use size_t in a few more places.

Declare many flag constants as unsigned to avoid sign mismatch
when shifting bits or applying the unary ~ operator.

When applying the unary ~ operator to enum constants, explictly
cast the result to an unsigned type, because enum constants can
be treated as signed.

In InnoDB, change the source code line number parameters from
ulint to unsigned type. Also, make some InnoDB functions return
a narrower type (unsigned or uint32_t instead of ulint;
bool instead of ibool).
2017-03-07 19:07:27 +02:00
Marko Mäkelä
5b07334b32 Remove an unused declaration.
The function thd_supports_xa() was removed in MySQL 5.7 in WL#8843.
2017-03-06 11:53:51 +02:00
Marko Mäkelä
7331b83eed MDEV-9282 follow-up: Remove an unused variable.
Remove the unused variable innodb_calling_exit which was added
in MySQL 5.7 in an attempt to avoid failures in other threads when
an I/O thread calls exit().

Since MDEV-9282, InnoDB or XtraDB MariaDB Server is not calling
exit() at all, and the variable innodb_calling_exit was never set.
2017-03-06 10:07:04 +02:00
Marko Mäkelä
27b9989d31 MDEV-12121 Introduce build option WITH_INNODB_AHI to disable innodb_adaptive_hash_index
The InnoDB adaptive hash index is sometimes degrading the performance of
InnoDB, and it is sometimes disabled to get more consistent performance.
We should have a compile-time option to disable the adaptive hash index.

Let us introduce two options:

OPTION(WITH_INNODB_AHI "Include innodb_adaptive_hash_index" ON)
OPTION(WITH_INNODB_ROOT_GUESS "Cache index root block descriptors" ON)

where WITH_INNODB_AHI always implies WITH_INNODB_ROOT_GUESS.

As part of this change, the misleadingly named function
trx_search_latch_release_if_reserved(trx) will be replaced with the macro
trx_assert_no_search_latch(trx) that will be empty unless
BTR_CUR_HASH_ADAPT is defined (cmake -DWITH_INNODB_AHI=ON).

We will also remove the unused column
INFORMATION_SCHEMA.INNODB_TRX.TRX_ADAPTIVE_HASH_TIMEOUT.
In MariaDB Server 10.1, it used to reflect the value of
trx_t::search_latch_timeout which could be adjusted during
row_search_for_mysql(). In 10.2, there is no such field.

Other than the removal of the unused column TRX_ADAPTIVE_HASH_TIMEOUT,
this is an almost non-functional change to the server when using the
default build options.

Some tests are adjusted so that they will work with both
-DWITH_INNODB_AHI=ON and -DWITH_INNODB_AHI=OFF. The test
innodb.innodb_monitor has been renamed to innodb.monitor
in order to track MySQL 5.7, and the duplicate tests
sys_vars.innodb_monitor_* are removed.
2017-03-03 16:55:50 +02:00
Marko Mäkelä
ab8199f38e MDEV-12103 Reduce the time of looking for MLOG_CHECKPOINT during crash recovery
This fixes MySQL Bug#80788 in MariaDB 10.2.5.

When I made the InnoDB crash recovery more robust by implementing
WL#7142, I also introduced an extra redo log scan pass that can be
shortened.

This fix will slightly extend the InnoDB redo log format that I
introduced in MySQL 5.7.9 by writing the start LSN of the MLOG_CHECKPOINT
mini-transaction to the end of the log checkpoint page, so that recovery
can jump straight to it without scanning all the preceding redo log.

LOG_CHECKPOINT_END_LSN: At the end of the checkpoint page, the start LSN
of the MLOG_CHECKPOINT mini-transaction. Previously, these bytes were
written as 0.

log_write_checkpoint_info(), log_group_checkpoint(): Add the parameter
end_lsn for writing LOG_CHECKPOINT_END_LSN.

log_checkpoint(): Remember the LSN at which the MLOG_CHECKPOINT
mini-transaction is starting (or at which the redo log ends on
shutdown).

recv_init_crash_recovery(): Remove.

recv_group_scan_log_recs(): Add the parameter checkpoint_lsn.

recv_recovery_from_checkpoint_start(): Read LOG_CHECKPOINT_END_LSN
and if it is set, start the first scan from it instead of the
checkpoint LSN. Improve some messages and remove bogus assertions.

recv_parse_log_recs(): Do not skip DBUG_PRINT("ib_log") for some
file-level redo log records.

recv_parse_or_apply_log_rec_body(): If we have not parsed all redo
log between the checkpoint and the corresponding MLOG_CHECKPOINT
record, defer the check for MLOG_FILE_DELETE or MLOG_FILE_NAME records
to recv_init_crash_recovery_spaces().

recv_init_crash_recovery_spaces(): Refuse recovery if
MLOG_FILE_NAME or MLOG_FILE_DELETE records are missing.
2017-03-03 09:38:59 +02:00
Marko Mäkelä
6cf29ab0df MDEV-12146 Deprecate and remove innodb_instrument_semaphores
MDEV-7618 introduced configuration parameter innodb_instrument_semaphores
in MariaDB Server 10.1. The parameter seems to only affect the rw-lock
X-latch acquisition. Extra fields are added to rw_lock_t to remember one
X-latch holder or waiter. These fields are not being consulted or reported
anywhere. This is basically only adding code bloat.

If the intention is to debug hangs or deadlocks, we have better tools for
that in the debug server, and for the non-debug server, core dumps can
reveal a lot. For example, the mini-transaction memo records the
currently held buffer block or index rw-locks, to be released at
mtr_t::commit().

The configuration parameter innodb_instrument_semaphores will be
deprecated in 10.2.5 and removed in 10.3.0.

rw_lock_t: Remove the members lock_name, file_name, line, thread_id
which did not affect any output.
2017-02-28 12:42:45 +02:00
Marko Mäkelä
c1bcb2055e Remove the unused field tablespace_version.
The field should have been removed when merging MySQL 5.7.9
to MariaDB 10.2.2 in commit 2e814d4702.
2017-02-27 13:44:53 +02:00
Marko Mäkelä
78153cf641 MDEV-11927 InnoDB change buffer is not being merged
to tables in the system tablespace

This is a regression caused by MDEV-11585, which accidentally
changed Tablespace::is_undo_tablespace() in an incorrect way,
causing the InnoDB system tablespace to be reported as a dedicated
undo tablespace, for which the change buffer is not applicable.

Tablespace::is_undo_tablespace(): Remove. There were only 2
calls from the function buf_page_io_complete(). Replace those
calls as appropriate.

Also, merge changes to tablespace import/export tests from
MySQL 5.7, and clean up the tests a little further, allowing
them to be run with any innodb_page_size.

Remove duplicated error injection instrumentation for the
import/export tests.  In MySQL 5.7, the error injection label
buf_page_is_corrupt_failure was renamed to
buf_page_import_corrupt_failure.

fil_space_extend_must_retry(): Correct a debug assertion
(tablespaces can be extended during IMPORT), and remove a
TODO comment about compressed temporary tables that was
already addressed in MDEV-11816.

dict_build_tablespace_for_table(): Correct a comment that
no longer holds after MDEV-11816, and assert that
ROW_FORMAT=COMPRESSED can only be used in .ibd files.
2017-02-24 22:16:33 +02:00
Marko Mäkelä
a13a636c74 MDEV-11802 innodb.innodb_bug14676111 fails
The function trx_purge_stop() was calling os_event_reset(purge_sys->event)
before calling rw_lock_x_lock(&purge_sys->latch). The os_event_set()
call in srv_purge_coordinator_suspend() is protected by that X-latch.

It would seem a good idea to consistently protect both os_event_set()
and os_event_reset() calls with a common mutex or rw-lock in those
cases where os_event_set() and os_event_reset() are used
like condition variables, tied to changes of shared state.

For each os_event_t, we try to document the mutex or rw-lock that is
being used. For some events, frequent calls to os_event_set() seem to
try to avoid hangs. Some events are never waited for infinitely, only
timed waits, and os_event_set() is used for early termination of these
waits.

os_aio_simulated_put_read_threads_to_sleep(): Define as a null macro
on other systems than Windows. TODO: remove this altogether and disable
innodb_use_native_aio on Windows.

os_aio_segment_wait_events[]: Initialize only if innodb_use_native_aio=0.

log_write_flush_to_disk_low(): Invoke log_mutex_enter() at the end, to
avoid race conditions when changing the system state. (No potential
race condition existed before MySQL 5.7.)
2017-02-20 12:32:43 +02:00
Marko Mäkelä
703d0985ee Fix some InnoDB memory leaks.
dict_init_free(): Make global, and move the call from
dict_close() to srv_free(), because this is initialized
earlier than dict_sys.

innobase_space_shutdown(): Do not leak srv_allow_writes_event.
2017-02-15 08:57:43 +02:00
Marko Mäkelä
2af28a363c MDEV-11782: Redefine the innodb_encrypt_log format
Write only one encryption key to the checkpoint page.
Use 4 bytes of nonce. Encrypt more of each redo log block,
only skipping the 4-byte field LOG_BLOCK_HDR_NO which the
initialization vector is derived from.

Issue notes, not warning messages for rewriting the redo log files.

recv_recovery_from_checkpoint_finish(): Do not generate any redo log,
because we must avoid that before rewriting the redo log files, or
otherwise a crash during a redo log rewrite (removing or adding
encryption) may end up making the database unrecoverable.
Instead, do these tasks in innobase_start_or_create_for_mysql().

Issue a firm "Missing MLOG_CHECKPOINT" error message. Remove some
unreachable code and duplicated error messages for log corruption.

LOG_HEADER_FORMAT_ENCRYPTED: A flag for identifying an encrypted redo
log format.

log_group_t::is_encrypted(), log_t::is_encrypted(): Determine
if the redo log is in encrypted format.

recv_find_max_checkpoint(): Interpret LOG_HEADER_FORMAT_ENCRYPTED.

srv_prepare_to_delete_redo_log_files(): Display NOTE messages about
adding or removing encryption. Do not issue warnings for redo log
resizing any more.

innobase_start_or_create_for_mysql(): Rebuild the redo logs also when
the encryption changes.

innodb_log_checksums_func_update(): Always use the CRC-32C checksum
if innodb_encrypt_log. If needed, issue a warning
that innodb_encrypt_log implies innodb_log_checksums.

log_group_write_buf(): Compute the checksum on the encrypted
block contents, so that transmission errors or incomplete blocks can be
detected without decrypting.

Rewrite most of the redo log encryption code. Only remember one
encryption key at a time (but remember up to 5 when upgrading from the
MariaDB 10.1 format.)
2017-02-15 08:07:20 +02:00
Marko Mäkelä
1b4b4f6887 MDEV-12057 Embedded server shutdown hangs in InnoDB
Ever since MDEV-5800 enabled indexed virtual columns for InnoDB,
the InnoDB shutdown relied on close_connections() that would set
thd->killed for the InnoDB purge threads. Alas, the embedded server
shutdown is not invoking close_connections(), and thus InnoDB purge
threads fail to initiate shutdown, causing a hang.

innodb_inited: Remove. Use srv_was_started instead.

innobase_fast_shutdown: Remove. Use srv_fast_shutdown instead.

srv_running: Renamed from thd_destructor_myvar, and made global.
The value NULL means that shutdown was requested or the purge threads
should not be running because of innodb_read_only_mode=1.

innobase_init(): Set srv_was_started after ensuring that srv_running
was initialized. (In innodb_read_only mode, the purge threads are not
started and we do not care if srv_running==NULL.)

innobase_start_or_create_for_mysql(): Do not set srv_was_started.
Let it be set by the only caller innobase_init().

srv_purge_should_exit(): Check also srv_was_started and srv_running
when evaluating thd->killed.
2017-02-14 10:12:38 +02:00
Marko Mäkelä
588eca31e3 Post-fix for MDEV-12050 Remove unused InnoDB Memcached hooks
Remove also the field trx_t::read_write that was only used by
the Memcached hooks.
2017-02-13 11:40:19 +02:00
Marko Mäkelä
a45866c6db MDEV-12050 Remove unused InnoDB Memcached hooks
Oracle introduced a Memcached plugin interface to the InnoDB
storage engine in MySQL 5.6. That interface is essentially a
fork of Memcached development snapshot 1.6.0-beta1 of an old
development branch 'engine-pu'.

To my knowledge, there have not been any updates to the Memcached code
between MySQL 5.6 and 5.7; only bug fixes and extensions related to
the Oracle modifications.

The Memcached plugin is not part of the MariaDB Server. Therefore it
does not make sense to include the InnoDB interfaces for the Memcached
plugin, or to have any related configuration parameters:

    innodb_api_bk_commit_interval
    innodb_api_disable_rowlock
    innodb_api_enable_binlog
    innodb_api_enable_mdl
    innodb_api_trx_level

Removing this code in one commit makes it possible to easily restore
it, in case it turns out to be needed later.
2017-02-13 11:24:02 +02:00
Marko Mäkelä
3272a19741 MDEV-11782 preparation: Add separate code for validating the 10.1 redo log.
log_crypt_101_read_checkpoint(): Read the encryption information
from a MariaDB 10.1 checkpoint page.

log_crypt_101_read_block(): Attempt to decrypt a MariaDB 10.1
redo log page.

recv_log_format_0_recover(): Only attempt decryption on checksum
mismatch. NOTE: With the MariaDB 10.1 innodb_encrypt_log format,
we can actually determine from the cleartext portion of the redo log
whether the redo log is empty. We do not really have to decrypt the
redo log here, if we did not want to determine if the checksum is valid.
2017-02-13 11:12:58 +02:00
Marko Mäkelä
96c4b9d49f MDEV-11782 preparation: Remove recv_sys_t::last_block.
We can use log_sys->buf instead of recv_sys->last_block during
crash recovery. Remove the redundant buffer.
2017-02-13 11:12:46 +02:00
Marko Mäkelä
070a8754c4 MDEV-12024 InnoDB startup fails to wait for recv_writer_thread to finish
recv_writer_thread(): Do not assign recv_writer_thread_active=true
in order to avoid a race condition with
recv_recovery_from_checkpoint_finish().

recv_init_crash_recovery_spaces(): Assign recv_writer_thread_active=true
before creating recv_writer_thread.
2017-02-08 15:42:15 +02:00
Marko Mäkelä
b40a1fbc93 MDEV-11782 WIP: Clean up the code, and add a test.
LOG_CHECKPOINT_ARRAY_END, LOG_CHECKPOINT_SIZE: Remove.

Change some error messages to refer to MariaDB 10.2.2 instead of
MySQL 5.7.9.

recv_find_max_checkpoint_0(): Do not abort when decrypting one of the
checkpoint pages fails.
2017-02-07 11:55:16 +02:00
Marko Mäkelä
bc12d993d7 MDEV-11947 InnoDB purge workers fail to shut down
srv_release_threads(): Actually wait for the threads to resume
from suspension. On CentOS 5 and possibly other platforms,
os_event_set() may be lost.

srv_resume_thread(): A counterpart of srv_suspend_thread().
Optionally wait for the event to be set, optionally with a timeout,
and then release the thread from suspension.

srv_free_slot(): Unconditionally suspend the thread. It is always
in resumed state when this function is entered.

srv_active_wake_master_thread_low(): Only call os_event_set().

srv_purge_coordinator_suspend(): Use srv_resume_thread() instead
of the complicated logic.
2017-02-03 12:52:37 +02:00
Marko Mäkelä
81b7fe9d38 Shut down InnoDB after aborted startup.
This fixes memory leaks in tests that cause InnoDB startup to fail.

buf_pool_free_instance(): Also free buf_pool->flush_rbt, which would
normally be freed when crash recovery finishes.

fil_node_close_file(), fil_space_free_low(), fil_close_all_files():
Relax some debug assertions to tolerate !srv_was_started.

innodb_shutdown(): Renamed from innobase_shutdown_for_mysql().
Changed the return type to void. Do not assume that all subsystems
were started.

que_init(), que_close(): Remove (empty functions).

srv_init(), srv_general_init(): Remove as global functions.

srv_free(): Allow srv_sys=NULL.

srv_get_active_thread_type(): Only return SRV_PURGE if purge really
is running.

srv_shutdown_all_bg_threads(): Do not reset srv_start_state. It will
be needed by innodb_shutdown().

innobase_start_or_create_for_mysql(): Always call srv_boot() so that
innodb_shutdown() can assume that it was called. Make more subsystems
dependent on SRV_START_STATE_STAT.

srv_shutdown_bg_undo_sources(): Require SRV_START_STATE_STAT.

trx_sys_close(): Do not assume purge_sys!=NULL. Do not call
buf_dblwr_free(), because the doublewrite buffer can exist while
the transaction system does not.

logs_empty_and_mark_files_at_shutdown(): Do a faster shutdown if
!srv_was_started.

recv_sys_close(): Invoke dblwr.pages.clear() which would normally
be invoked by buf_dblwr_process().

recv_recovery_from_checkpoint_start(): Always release log_sys->mutex.

row_mysql_close(): Allow the subsystem not to exist.
2017-02-01 09:30:55 +02:00
Marko Mäkelä
16bc16f9ba Actually invoke free() in ut_allocator::deallocate().
The necessary call was inadvertently commented out in a
merge of MySQL 5.7.14 to MariaDB 10.2.1
(commit fec844aca8).
2017-02-01 09:30:55 +02:00
Marko Mäkelä
1293e5e59b Rewrite the innodb.log_file_size test with DBUG_EXECUTE_IF.
Remove the debug parameter innodb_force_recovery_crash that was
introduced into MySQL 5.6 by me in WL#6494 which allowed InnoDB
to resize the redo log on startup.

Let innodb.log_file_size actually start up the server, but ensure
that the InnoDB storage engine refuses to start up in each of the
scenarios.
2017-01-31 10:23:20 +02:00
Marko Mäkelä
3271da11c4 MDEV-11814 Refuse innodb_read_only startup if crash recovery is needed
recv_scan_log_recs(): Remember if redo log apply is needed,
even if starting up in innodb_read_only mode.

recv_recovery_from_checkpoint_start_func(): Refuse
innodb_read_only startup if redo log apply is needed.
2017-01-27 13:01:56 +02:00
Jan Lindström
71495a1748 MDEV-11849: Fix storage/innobase/* compile warnings 2017-01-26 08:05:24 +02:00
Marko Mäkelä
0b1a40852c Replace fil_node_t::is_open with fil_node_t::is_open().
This should be a non-functional change (apart from removing the
redundant data field).
2017-01-25 16:00:34 +02:00
Jan Lindström
8531e19771 buf_page_t is class on 10.2 not struct. 2017-01-25 12:33:39 +02:00
Jan Lindström
4b28798f95 Fix compiler error on x86. 2017-01-25 10:04:14 +02:00
Jan Lindström
84895c3cd7 Fix compiler error on x86. 2017-01-25 08:54:02 +02:00
Jan Lindström
6495806e59 MDEV-11254: innodb-use-trim has no effect in 10.2
Problem was that implementation merged from 10.1 was incompatible
with InnoDB 5.7.

buf0buf.cc: Add functions to return should we punch hole and
how big.

buf0flu.cc: Add written page to IORequest

fil0fil.cc: Remove unneeded status call and add test is
sparse files and punch hole supported by file system when
tablespace is created. Add call to get file system
block size. Used file node is added to IORequest. Added
functions to check is punch hole supported and setting
punch hole.

ha_innodb.cc: Remove unneeded status variables (trim512-32768)
and trim_op_saved. Deprecate innodb_use_trim and
set it ON by default. Add function to set innodb-use-trim
dynamically.

dberr.h: Add error code DB_IO_NO_PUNCH_HOLE
if punch hole operation fails.

fil0fil.h: Add punch_hole variable to fil_space_t and
block size to fil_node_t.

os0api.h: Header to helper functions on buf0buf.cc and
fil0fil.cc for os0file.h

os0file.h: Remove unneeded m_block_size from IORequest
and add bpage to IORequest to know actual size of
the block and m_fil_node to know tablespace file
system block size and does it support punch hole.

os0file.cc: Add function punch_hole() to IORequest
to do punch_hole operation,
get the file system block size and determine
does file system support sparse files (for punch hole).

page0size.h: remove implicit copy disable and
use this implicit copy to implement copy_from()
function.

buf0dblwr.cc, buf0flu.cc, buf0rea.cc, fil0fil.cc, fil0fil.h,
os0file.h, os0file.cc, log0log.cc, log0recv.cc:
Remove unneeded write_size parameter from fil_io
calls.

srv0mon.h, srv0srv.h, srv0mon.cc: Remove unneeded
trim512-trim32678 status variables. Removed
these from monitor tests.
2017-01-24 14:40:58 +02:00
Marko Mäkelä
b05bf8ff0f Merge 10.1 to 10.2.
Most notably, this includes MDEV-11623, which includes a fix and
an upgrade procedure for the InnoDB file format incompatibility
that is present in MariaDB Server 10.1.0 through 10.1.20.

In other words, this merge should address
MDEV-11202 InnoDB 10.1 -> 10.2 migration does not work
2017-01-19 12:06:13 +02:00
Marko Mäkelä
a1315a650a MDEV-11202 InnoDB 10.1 -> 10.2 migration does not work
This is the final preparation for the merge of MDEV-11623
from MariaDB Server 10.1 (correcting the assignment of
FSP_SPACE_FLAGS in MariaDB Server 10.1).

We must avoid reading FSP_SPACE_FLAGS directly from page 0,
because the flags that affect the uncompressed page size
cannot be trusted if we are upgrading from MariaDB 10.1.

We will pass fil_space_t* instead of a numeric tablespace ID
in many low-level functions, so that fil_space_t::flags
will be available. This will also reduce the amount o
tablespace ID lookups and the pressure on fil_system->mutex.

fil_space_is_being_truncated(), fil_space_is_redo_skipped(),
PageCallback::set_page_size(), fsp_header_get_page_size(): Remove.

fil_node_open_file(), fil_space_get_crypt_data(): Use space->flags.

fsp_free_extent(), fseg_get_first_extent(), fsp_get_space_header(),
xdes_get_descriptor_with_space_hdr(), xdes_get_descriptor(),
xdes_lst_get_descriptor(), fsp_space_modify_check(),
fsp_init_file_page(), fsp_alloc_free_extent(), fsp_page_create(),
fsp_alloc_free_page(), fsp_free_page(), fsp_alloc_seg_inode_page(),
fsp_alloc_seg_inode(), fsp_free_seg_inode(), fseg_fill_free_list(),
fseg_mark_page_used(), fseg_free_page_low(), fseg_free_extent():
Take fil_space_t* as a parameter, instead of taking a numeric ID.
2017-01-18 14:57:10 +02:00
Marko Mäkelä
8780b89529 MDEV-11831 Make InnoDB mini-transaction memo checks stricter
InnoDB keeps track of buffer-fixed buf_block_t or acquired rw_lock_t
within a mini-transaction. There are some memo_contains assertions
in the code that document when certain blocks or rw_locks must be held.
But, these assertions only check the mini-transaction memo, not the fact
whether the rw_lock_t are actually being held by the caller.

btr_pcur_store_position(): Remove #ifdef, and assert that the block
is always buffer-fixed.

rtr_pcur_getnext_from_path(), rtr_pcur_open_low(),
ibuf_rec_get_page_no_func(), ibuf_rec_get_space_func(),
ibuf_rec_get_info_func(), ibuf_rec_get_op_type_func(),
ibuf_build_entry_from_ibuf_rec_func(), ibuf_rec_get_volume_func(),
ibuf_get_merge_page_nos_func(), ibuf_get_volume_buffered_count_func()
ibuf_get_entry_counter_low_func(), page_set_ssn_id(),
row_vers_old_has_index_entry(), row_vers_build_for_consistent_read(),
row_vers_build_for_semi_consistent_read(),
trx_undo_prev_version_build():
Make use of mtr_memo_contains_page_flagged().

mtr_t::memo_contains(): Take a const memo. Assert rw_lock_own().

FindPage, FlaggedCheck: Assert rw_lock_own_flagged().
2017-01-18 14:57:10 +02:00