Commit graph

4385 commits

Author SHA1 Message Date
Marko Mäkelä
6c91be54b1 MDEV-11520 Properly retry posix_fallocate() on EINTR
We only want to retry posix_fallocate() on EINTR as long as the system
is not being shut down. We do not want to retry on any other (hard) error.

Thanks to Jocelyn Fournier for quickly noticing the mistake in my
previous commit.
2017-05-06 15:50:09 +03:00
Marko Mäkelä
14c6f00a9f Merge 10.1 into 10.2
Also, include fixes by Vladislav Vaintroub to the
aws_key_management plugin. The AWS C++ SDK specifically depends on
OPENSSL_LIBRARIES, not generic SSL_LIBRARIES (such as YaSSL).
2017-05-06 14:36:46 +03:00
Marko Mäkelä
baad0f3484 MDEV-11520 post-fix: Retry posix_fallocate() on EINTR in fil_ibd_create()
Earlier versions of MariaDB only use posix_fallocate() when extending
data files, not when initially creating the files,
2017-05-06 14:33:14 +03:00
Oleksandr Byelkin
82811f7966 MDEV-12485 foreign key on delete cascade stale entries with query cache enabled
During merge of innodb code QC invalidation was removed from innodb part but not added to server part.
We decided to keep it in innodb to keep server/engine interface the same.
2017-05-05 10:13:42 +02:00
Marko Mäkelä
e946297d8f Remove an unused variable. 2017-05-05 11:06:46 +03:00
Marko Mäkelä
f9cc391863 Merge 10.1 into 10.2
This only merges MDEV-12253, adapting it to MDEV-12602 which is already
present in 10.2 but not yet in the 10.1 revision that is being merged.

TODO: Error handling in crash recovery needs to be improved.
If a page cannot be decrypted (or read), we should cleanly abort
the startup. If innodb_force_recovery is specified, we should
ignore the problematic page and apply redo log to other pages.
Currently, the test encryption.innodb-redo-badkey randomly fails
like this (the last messages are from cmake -DWITH_ASAN):

2017-05-05 10:19:40 140037071685504 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1635994
2017-05-05 10:19:40 140037071685504 [ERROR] InnoDB: Missing MLOG_FILE_NAME or MLOG_FILE_DELETE before MLOG_CHECKPOINT for tablespace 1
2017-05-05 10:19:40 140037071685504 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[2201] with error Data structure corruption
2017-05-05 10:19:41 140037071685504 [Note] InnoDB: Starting shutdown...
i=================================================================
==5226==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x612000018588 in thread T0
    #0 0x736750 in operator delete(void*) (/mariadb/server/build/sql/mysqld+0x736750)
    #1 0x1e4833f in LatchCounter::~LatchCounter() /mariadb/server/storage/innobase/include/sync0types.h:599:4
    #2 0x1e480b8 in LatchMeta<LatchCounter>::~LatchMeta() /mariadb/server/storage/innobase/include/sync0types.h:786:17
    #3 0x1e35509 in sync_latch_meta_destroy() /mariadb/server/storage/innobase/sync/sync0debug.cc:1622:3
    #4 0x1e35314 in sync_check_close() /mariadb/server/storage/innobase/sync/sync0debug.cc:1839:2
    #5 0x1dfdc18 in innodb_shutdown() /mariadb/server/storage/innobase/srv/srv0start.cc:2888:2
    #6 0x197e5e6 in innobase_init(void*) /mariadb/server/storage/innobase/handler/ha_innodb.cc:4475:3
2017-05-05 10:38:53 +03:00
Sergei Golubchik
0072d2e9a1 InnoDB cleanup: remove a bunch of #ifdef UNIV_INNOCHECKSUM
innochecksum uses global variables. great, let's use them all the
way down, instead of passing them as arguments to innodb internals,
conditionally modifying function prototypes with #ifdefs
2017-04-30 14:58:11 +02:00
Marko Mäkelä
b82c602db5 MDEV-12602 InnoDB: Failing assertion: space->n_pending_ops == 0
This fixes a regression caused by MDEV-12428.
When we introduced a variant of fil_space_acquire() that could
increment space->n_pending_ops after space->stop_new_ops was set,
the logic of fil_check_pending_operations() was broken.

fil_space_t::n_pending_ios: A new field to track read or write
access from the buffer pool routines immediately before a block
write or after a block read in the file system.

fil_space_acquire_for_io(), fil_space_release_for_io(): Similar
to fil_space_acquire_silent() and fil_space_release(), but
modify fil_space_t::n_pending_ios instead of fil_space_t::n_pending_ops.

Adjust a number of places accordingly, and remove some redundant
tablespace lookups.

The following parts of this fix differ from the 10.2 version of this fix:

buf_page_get_corrupt(): Add a tablespace parameter.

In 10.2, we already had a two-phase process of freeing fil_space objects
(first, fil_space_detach(), then release fil_system->mutex, and finally
free the fil_space and fil_node objects).

fil_space_free_and_mutex_exit(): Renamed from fil_space_free().
Detach the tablespace from the fil_system cache, release the
fil_system->mutex, and then wait for space->n_pending_ios to reach 0,
to avoid accessing freed data in a concurrent thread.
During the wait, future calls to fil_space_acquire_for_io() will
not find this tablespace, and the count can only be decremented to 0,
at which point it is safe to free the objects.

fil_node_free_part1(), fil_node_free_part2(): Refactored from
fil_node_free().
2017-04-28 14:12:52 +03:00
Marko Mäkelä
4b24467ff3 MDEV-12602 InnoDB: Failing assertion: space->n_pending_ops == 0
This fixes a regression caused by MDEV-12428.
When we introduced a variant of fil_space_acquire() that could
increment space->n_pending_ops after space->stop_new_ops was set,
the logic of fil_check_pending_operations() was broken.

fil_space_t::n_pending_ios: A new field to track read or write
access from the buffer pool routines immediately before a block
write or after a block read in the file system.

fil_space_acquire_for_io(), fil_space_release_for_io(): Similar
to fil_space_acquire_silent() and fil_space_release(), but
modify fil_space_t::n_pending_ios instead of fil_space_t::n_pending_ops.

fil_space_free_low(): Wait for space->n_pending_ios to reach 0,
to avoid accessing freed data in a concurrent thread. Future
calls to fil_space_acquire_for_io() will not find this tablespace,
because it will already have been detached from fil_system.

Adjust a number of places accordingly, and remove some redundant
tablespace lookups.

FIXME: buf_page_check_corrupt() should take a tablespace from
fil_space_acquire_for_io() as a parameter. This will be done
in the 10.1 version of this patch and merged from there.
That depends on MDEV-12253, which has not been merged from 10.1 yet.
2017-04-28 12:23:35 +03:00
Marko Mäkelä
f740d23ce6 Merge 10.1 into 10.2 2017-04-28 12:22:32 +03:00
Sergei Golubchik
6935d66053 MDEV-12591 MariaDB embedded server refuses start with innodb_plugin
innodb calls sd_notify(), it must be linked with libsystemd
2017-04-27 19:12:45 +02:00
Vladislav Vaintroub
db39107413 MDEV-11663 Create services for functionality used by plugins
Added service for
- encryption (AES)
- error reporting, e.g my_printf_error()
2017-04-27 19:12:38 +02:00
Aditya A
14fe6dd239 Bug #23046302 COUNT(*) MUCH SLOWER ON 5.7 THAN 5.6
ANALYSIS

This is regression caused due to worklog 6742 which
implemented ha_innobase::records() which always
uses clustered index to get the row count. Previously
optimizer chose secondary index which was smaller in
size of clustered index to scan for rows and resulted in
a quicker scan.

FIX

After discussion it was decided to remove this feature in 5.7.

[#rb14040 Approved by Kevin and Oystein ]
2017-04-26 23:03:33 +03:00
Thirunarayanan Balathandayuthapani
fca0698fd8 Bug #25385590 DROP TABLE CRASHES IF INNODB_FORCE_RECOVERY > 4
- Allow the server to start if innodb force recovery is set to 6
even though change buffer is not empty

Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>
2017-04-26 23:03:33 +03:00
Marko Mäkelä
67e9c4cf6c Adapt the test case for Oracle Bug#25385590
buf_chunk_not_freed(), logs_empty_and_mark_files_at_shutdown():
Relax debug assertions when innodb_force_recovery=6
implies innodb_read_only.
2017-04-26 23:03:33 +03:00
Thirunarayanan Balathandayuthapani
7223ec4ca7 Bug #25385590 DROP TABLE CRASHES IF INNODB_FORCE_RECOVERY > 4
Problem:
========
 - Drop table assert if innodb_force_recovery is set to 5 or 6.
For innodb_force_recovery 5 and 6, InnoDB doesn't scan the undo log
and it makes the redo rollback segment as NULL. There is no way for
transaction to write any undo log.

 - If innodb_force_recovery is set to 6 then InnoDB does not do the
redo log roll-forward in connection with recovery. In this case,
log_sys will be initalized only and it will not have latest
checkpoint information. Checkpoint is done during shutdown even
innodb_force_recovery is set to 6. So it leads to incorrect
information update in checkpoint header.

Solution:
========
1) Allow drop table only if innodb_force_recovery < 5.
2) Make innodb as read-only if innodb_force_recovery is set to 6.
3) During shutdown, remove the checkpoint if innodb_force_recovery
is set to 6.

Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>
RB: 15075
2017-04-26 23:03:33 +03:00
Thirunarayanan Balathandayuthapani
2ef1baa75f Bug #24793413 LOG PARSING BUFFER OVERFLOW
Problem:
========
During checkpoint, we are writing all MLOG_FILE_NAME records in one mtr
and parse buffer can't be processed till MLOG_MULTI_REC_END. Eventually parse
buffer exceeds the RECV_PARSING_BUF_SIZE and eventually it overflows.

Fix:
===
1) Break the large mtr if it exceeds LOG_CHECKPOINT_FREE_PER_THREAD into multiple mtr during checkpoint.
2) Move the parsing buffer if we are encountering only MLOG_FILE_NAME
records. So that it will never exceed the RECV_PARSING_BUF_SIZE.

Reviewed-by: Debarun Bannerjee <debarun.bannerjee@oracle.com>
Reviewed-by: Rahul M Malik <rahul.m.malik@oracle.com>
RB: 14743
2017-04-26 23:03:32 +03:00
Thirunarayanan Balathandayuthapani
88a84f49b3 Bug #25167032 CRASH WHEN ASSIGNING MY_ERRNO - MISSING MY_THREAD_INIT IN BACKGROUND THREAD
Description:
===========
Add my_thread_init() and my_thread_exit() for background threads which
initializes and frees the st_my_thread_var structure.

Reviewed-by: Jimmy Yang<jimmy.yang@oracle.com>
RB: 15003
2017-04-26 23:03:32 +03:00
rahul malik
dbe4c4e354 BUG#25330449 ASSERT SIZE==SPACE->SIZE DURING BUF_READ_AHEAD_RANDOM
Problem:

During read head, wrong page size is used to calcuate the tablespace size.

Fix:

Use physical page size to calculate tablespace size

Reveiwed-By: Satya Bodapati
RB: 14993
2017-04-26 23:03:32 +03:00
Marko Mäkelä
849af74a48 MariaDB adjustments for Oracle Bug#23070734 fix
Split the test case so that a server restart is not needed.
Reduce the test cases and use a simpler mechanism for triggering
and waiting for purge.

fil_table_accessible(): Check if a table can be accessed without
enjoying MDL protection.
2017-04-26 23:03:32 +03:00
Aditya A
62dca454e7 Bug #23070734 CONCURRENT TRUNCATE TABLES CAUSE STALLS
PROBLEM

When truncating single tablespace tables, we need to scan the entire
buffer pool to remove the pages of the table from the buffer pool.
During this scan and removal dict_sys->mutex is being held ,causing
stalls in other DDL operations.

FIX

Release the dict_sys->mutex during the scan and reacquire it after the
scan. Make sure that purge thread doesn't purge the records of the table
being truncated and background stats collection thread skips the updation
of stats for the table being truncated.

[#rb 14564 Approved by Jimmy and satya ]
2017-04-26 23:03:31 +03:00
Marko Mäkelä
13dcdb0903 MDEV-11802 InnoDB purge does not always run when there is work to do
srv_sys_t::n_threads_active[]: Protect writes by both the mutex and
by atomic memory access.

srv_active_wake_master_thread_low(): Reliably wake up the master
thread if there is work to do. The trick is to atomically read
srv_sys->n_threads_active[].

srv_wake_purge_thread_if_not_active(): Atomically read
srv_sys->n_threads_active[] (and trx_sys->rseg_history_len),
so that the purge should always be triggered when there is work to do.

trx_commit_in_memory(): Invoke srv_wake_purge_thread_if_not_active()
whenever a transaction is committed. Purge could have been prevented by
the read view of the currently committing transaction, even if it is
a read-only transaction.

trx_purge_add_update_undo_to_history(): Do not wake up the purge.
This is only called by trx_undo_update_cleanup(), as part of
trx_write_serialisation_history(), which in turn is only called by
trx_commit_low() which will always call trx_commit_in_memory().
Thus, the added call in trx_commit_in_memory() will cover also
this use case where a committing read-write transaction added
some update_undo log to the purge queue.

trx_rseg_mem_restore(): Atomically modify trx_sys->rseg_history_len.
2017-04-26 23:03:31 +03:00
Marko Mäkelä
731435af9a srv_purge_coordinator_thread(): Remove redundant variables 2017-04-26 23:03:31 +03:00
Marko Mäkelä
6264e892cf Remove redundant initialization of global InnoDB variables
Also, remove the unused global variable srv_priority_boost.
2017-04-26 23:03:31 +03:00
Marko Mäkelä
797db28d53 Remove the unused declaration fsp_is_temporary() 2017-04-26 23:03:31 +03:00
Thirunarayanan Balathandayuthapani
6b63e4ad4a Bug #23219499 CONCURRENT UPDATE DOESN'T APPLY IN VIRTUAL INDEX DURING TABLE REBUILD
Problem:
=======
Concurrent update dml statement doesn't reflect in virtual index during
inplace table rebuild. It results mismatch value in virutal index and
clustered index. Deleting the table content tries to search the mismatch
value in virtual index but it can't find the value. During log update
apply phase, virtual information is being ignored while constructing
the new entry.

Solution:
=========
In row_log_update_apply phase, build the entry with virtual column
information. So that it can reflect in newly constructed virtual index.

Reviewed-by: Jimmy Yang<jimmy.yang@oracle.com>
RB: 14974
2017-04-26 23:03:31 +03:00
Darshan M N
698e37d60c BUG#25251082 DISABLING CERTAIN MACROS IN INNODB RESULTS IN COMPILATION ERRORS
Issue:
======
Disabling macros such as UNIV_PFS_MUTEX/UNIV_PFS_RWLOCK/UNIV_PFS_THREAD
which are defined in InnoDB throws errors during compilation.

Fix:
====
Fix all the compilation errors.

RB: 14893
Reviewed-by: Jimmy Yang <Jimmy.Yang@oracle.com>
Reviewed-by: Satya Bodapati <satya.bodapati@oracle.com>
2017-04-26 23:03:30 +03:00
Darshan M N
16ed1f9c31 BUG#25053705 INVALID I/O ON TABLE AFTER TRUNCATE
Issue:
======
The issue is that if a fts index is present in a table the space size is
incorrectly calculated in the case of truncate which results in a invalid
read.

Fix:
====
Have a different space size calculation in truncate if fts indexes are
present.

RB:14755
Reviewed-by: Shaohua Wang <shaohua.wang@oracle.com>
2017-04-26 23:03:30 +03:00
Thirunarayanan Balathandayuthapani
c2d9c0ce85 Bug #24585978 INNODB: ASSERTION TOTAL_RECS > 0 FAILURE IN FILE DICT0STATS.CC
Analysis:
========
 There was missing bracket for IF conditon in dict_stats_analyze_index_level()
and it leads to wrong result.

Fix:
====
 Fix the IF condition in dict_stats_analyze_index_level() so that it satisfied
the if condtion only if level is zero.

Reviewed-by : Jimmy Yang <jimmy.yang@oracle.com>
2017-04-26 23:03:30 +03:00
Thirunarayanan Balathandayuthapani
35652ed631 Bug#25222337 FIELD NAME IS NULL IN NEWLY ADDED VIRTUAL INDEX FOR NEWLY ADDED VIRTUAL COLUMN
Analysis:
========
Field name comparison happens while filling the virtual columns
affected by foreign constraint. But field name is NULL in virtual
index for the newly added virtual column.

Fix:
===
Ignore the index if it has newly added virtual column. Foreign
key affected virtual column information is filled during
loading operation.

Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>
RB: 14895
2017-04-26 23:03:30 +03:00
Debarun Banerjee
49edf2d476 BUG#25126722 FOREIGN KEY CONSTRAINT NAME IS NULL IN INFORMATION_SCHEMA AFTER RESTART
Problem :
---------
Information_Schema.referential_constraints (UNIQUE_CONSTRAINT_NAME)
shows NULL for a foreign key constraint after restarting the server.
If any dml or query (select/insert/update/delete) is done on
referenced table, then the constraint name is correctly shown.

Solution :
----------
UNIQUE_CONSTRAINT_NAME column is the key name of the referenced table.
In innodb, FK reference is stored as a list of columns in referenced
table in INNODB_SYS_FOREIGN and INNODB_SYS_FOREIGN_COLS. The referenced
column must have at least one index/key with the referenced column as
prefix but the key name itself is not included in FK metadata. For this
reason, the UNIQUE_CONSTRAINT_NAME is only filled up when the
referenced table is actually loaded in innodb dictionary cache.

The information_schema view calls handler::get_foreign_key_list() on
foreign key table to read the FK metadata. The UNIQUE_CONSTRAINT_NAME
information shows NULL based on whether the referenced table is
already loaded or not.

One way to fix this issue is to load the referenced table while reading
the FK metadata information, if needed.

Reviewed-by: Sunny Bains <sunny.bains@oracle.com>

RB: 14654
2017-04-26 23:03:30 +03:00
Allen Lai
07e88be5b7 Bug#23044098 INSERT OF GIS DATA INTO RTREE HITS ASSERT IN RTR_CUR_RESTORE_POSITION_FUNC()
This bug is caused by missing page number field when store btree cursor.

Reviewed-by: Jimmy Yang<jimmy.yang@oracle.com>
RB: 14617
2017-04-26 23:03:30 +03:00
Debarun Banerjee
4e41ac26f5 BUG#25082593 FOREIGN KEY VALIDATION DOESN'T NEED TO ACQUIRE GAP LOCK IN READ COMMITTED
Problem :
---------
This bug is filed from the base replication bug#25040331 where the
slave thread times out while INSERT operation waits on GAP lock taken
during Foreign Key validation.

The primary reason for the lock wait is because the statements are
getting replayed in different order. However, we also observed
two things ...

1. The slave thread could always use "Read Committed" isolation for
row level replication.

2. It is not necessary to have GAP locks in "READ Committed" isolation
level in innodb.

This bug is filed to address point(2) to avoid taking GAP locks during
Foreign Key validation.

Solution :
----------
Innodb is primarily designed for "Repeatable Read" and the GAP lock
behaviour is default. For "Read Committed" isolation, we have special
handling in row_search_mvcc to avoid taking the GAP lock while
scanning records.

While looking for Foreign Key, the code is following the default
behaviour taking GAP locks. The suggested fix is to avoid GAP
locking during FK validation similar to normal search operation
(row_search_mvcc) for "Read Committed" isolation level.

Reviewed-by: Sunny Bains <sunny.bains@oracle.com>

RB: 14526
2017-04-26 23:03:29 +03:00
Debarun Banerjee
bf5be32376 BUG#25032066 PREPARED TRANSACTION SHOULD NOT BE ROLLED BACK BY HIGH PRIORITY TRANSACTION
Problem :
---------
1. delete_all_rows() and rnd_init() are not returning error
after async rollback in 5.7. This results in assert in
innodb in next call.

2. High priority transaction is rolling back prepared transaction.
This is because TRX_FORCE_ROLLBACK_DISABLE is getting set only for
first entry [TrxInInnoDB].

Solution :
----------
1. return DB_FORCED_ABORT error after rollback.
2. check and disable rollback in TrxInInnodb::enter always.

Reviewed-by: Sunny Bains <sunny.bains@oracle.com>

RB: 13777
2017-04-26 23:03:29 +03:00
Knut Anders Hatlen
9df0426103 Bug#25048573: STD::MAP INSTANTIATIONS CAUSE STATIC ASSERT FAILURES ON FREEBSD 11
Problem: Some instantiations of std::map have discrepancies between
the value_type of the map and the value_type of the map's allocator.
On FreeBSD 11 this is detected by Clang, and an error is raised at
compilation time.

Fix: Specify the correct value_type for the allocators.

Also fix an unused variable warning in storage/innobase/os/os0file.cc.
2017-04-26 23:03:29 +03:00
Vasil Dimov
8923f6b741 Fix Bug#24707869 GCC 5 AND 6 MISCOMPILE MACH_PARSE_COMPRESSED
Prevent GCC from moving a mach_read_from_4() before we have checked that
we have 4 bytes to read. The pointer may only point to a 1, 2 or 3
bytes in which case the code should not read 4 bytes. This is a
workaround to a GCC bug:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77673

Patch submitted by: Laurynas Biveinis <laurynas.biveinis@gmail.com>
RB: 14135
Reviewed by: Pawel Olchawa <pawel.olchawa@oracle.com>
2017-04-26 23:03:29 +03:00
Daniel Blanchard
32f99b288b Bug #24711351 64 BIT WINDOWS MYSQLD BUILD REPORTS INNODB: OPERATING SYSTEM ERROR NUMBER 995
Description
===========
Under heavy load, the aysnchronous Windows file IO API can return a
failure code that is handled in MySQL Server by retrying the file IO operation.

A cast necessary for the correct operation of the retry path in a 64 bit
build is missing, leading to the file IO retry result being misinterpreted
and ultimately the report of the OS error number 995
(ERROR_OPERATION_ABORTED) in the MySQL error log.

Fix
===
Supply the missing cast.

Reviewed-by: Sunny Bains <sunny.bains@oracle.com>
RB: 14109
2017-04-26 23:03:29 +03:00
Vasil Dimov
6c76e5a00b Fix Bug#24605956 SERVER MAY CRASH DUE TO A GLIBC BUG IN HANDLING SHORT-LIVED DETACHED THREADS
Avoid detaching and exiting from threads that may finish before the
caller has returned from pthread_create(). Only exit from such threads,
without detach and join with them later.

Patch submitted by: Laurynas Biveinis <laurynas.biveinis@gmail.com>
RB: 13983
Reviewed by: Sunny Bains <sunny.bains@oracle.com>
2017-04-26 23:03:29 +03:00
Marko Mäkelä
da76c1bd3e Minor cleanup 2017-04-26 23:03:28 +03:00
Marko Mäkelä
ce3ffefc45 Adapt the innodb_undo tests from MySQL 5.7
Simplify the tests that are present in MySQL 5.7. Make the table
smaller while generating enough undo log. Do not unnecessarily
drop tables.

trx_purge_initiate_truncate(): Remove two crash injection points
(before and after normal redo log checkpoint), because they are
not adding any value. Clarify some messages.

trx_sys_create_rsegs(): Display the number of active undo tablespaces.

srv_undo_tablespaces_init(): When initializing the data files, do not
leave srv_undo_tablespaces_active at 0.
Do not display that number; let trx_sys_create_rsegs() display it once
the final number is known.

innodb_params_adjust(): Adjust parameters after startup.

innobase_init(): Do not allow innodb_max_undo_size to be less
than SRV_UNDO_TABLESPACE_SIZE_IN_PAGES. This avoids unnecessary
repeated truncation of undo tablespaces when using
innodb_page_size=32k or innodb_page_size=64k.
2017-04-26 23:03:28 +03:00
Thirunarayanan Balathandayuthapani
d1bcc1f49f Bug #24488141 ACTIVE UNDO TABLESPACE NOT UPDATED WHEN INNODB_UNDO_LOGS IS INCREASED
Problem:
========
 If we increase innodb_undo_logs value during startup. New rollback
segment might get allocated during trx_sys_create_rseg(). Essentially, it
would make these tablesapces active with transaction undo data and purge.
But it doesn't come in active undo tablespaces. so these tablespace would
never get truncated.

Fix:
===
 Increase the number of active undo tablespace when we are assigning the
undo tablespace to the newly assigned rollback segment.

Reviewed-by: Kevin Lewis <kevin.lewis@oracle.com>
Reviewed-by: Debarun Banerjee <debarun.banerjee@oracle.com>
RB: 13746
2017-04-26 23:03:28 +03:00
Marko Mäkelä
bdfa49f6e2 Remove redundant initialization of some InnoDB startup parameters
The InnoDB startup parameters will be initialized via pointers
in innobase_system_variables[]. Remove some redundant link-time
initialization.
2017-04-26 23:03:28 +03:00
Marko Mäkelä
206ecb79a5 Follow-up to MDEV-12289: Support innodb_undo_tablespaces=127
MySQL 5.7 reduced the maximum number of innodb_undo_tablespaces
from 126 to 95 when it reserved 32 persistent rollback segments
for the temporary undo logs. Since MDEV-12289 restored all 128
persistent rollback segments for persistent undo logs, the
reasonable maximum value of innodb_undo_tablespaces is 127
(not 126 or 95). This is because out of the 128 rollback segments,
the first one will always be created in the system tablespace
and the remaining ones can be created in dedicated undo tablespaces.
2017-04-26 23:03:28 +03:00
Marko Mäkelä
23ea4360fd Remove TRX_SYS_OLD_N_RSEGS 2017-04-26 23:03:27 +03:00
Marko Mäkelä
472b5f0d1f Follow-up to Bug#24346574 PAGE CLEANER THREAD, ASSERT BLOCK->N_POINTERS == 0
Silence the Valgrind warnings on instrumented builds (-DWITH_VALGRIND).

assert_block_ahi_empty_on_init(): A variant of
assert_block_ahi_empty() that declares n_pointers initialized and then
asserts that n_pointers==0.

In Valgrind-instrumented builds, InnoDB declares allocated memory
uninitialized.
2017-04-26 23:03:27 +03:00
Marko Mäkelä
e63ead68bf Bug#24346574 PAGE CLEANER THREAD, ASSERT BLOCK->N_POINTERS == 0
btr_search_drop_page_hash_index(): Do not return before ensuring
that block->index=NULL, even if !btr_search_enabled. We would
typically still skip acquiring the AHI latch when the AHI is
disabled, because block->index would already be NULL. Only if the AHI
is in the process of being disabled, we would wait for the AHI latch
and then notice that block->index=NULL and return.

The above bug was a regression caused in MySQL 5.7.9 by the fix of
Bug#21407023: DISABLING AHI SHOULD AVOID TAKING AHI LATCH

The rest of this patch improves diagnostics by adding assertions.

assert_block_ahi_valid(): A debug predicate for checking that
block->n_pointers!=0 implies block->index!=NULL.

assert_block_ahi_empty(): A debug predicate for checking that
block->n_pointers==0.

buf_block_init(): Instead of assigning block->n_pointers=0,
assert_block_ahi_empty(block).

buf_pool_clear_hash_index(): Clarify comments, and assign
block->n_pointers=0 before assigning block->index=NULL.
The wrong ordering could make block->n_pointers appear incorrect in
debug assertions. This bug was introduced in MySQL 5.1.52 by
Bug#13006367 62487: INNODB TAKES 3 MINUTES TO CLEAN UP THE
ADAPTIVE HASH INDEX AT SHUTDOWN

i_s_innodb_buffer_page_get_info(): Add a comment that
the IS_HASHED column in the INFORMATION_SCHEMA views
INNODB_BUFFER_POOL_PAGE and INNODB_BUFFER_PAGE_LRU may
show false positives (there may be no pointers after all.)

ha_insert_for_fold_func(), ha_delete_hash_node(),
ha_search_and_update_if_found_func(): Use atomics for
updating buf_block_t::n_pointers. While buf_block_t::index is
always protected by btr_search_x_lock(index), in
ha_insert_for_fold_func() the n_pointers-- may belong to
another dict_index_t whose btr_search_latches[] we are not holding.

RB: 13879
Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>
2017-04-26 23:03:27 +03:00
Thirunarayanan Balathandayuthapani
a6adf567fd Bug #23533396 ASSERTION !M_PREBUILT->TRX->CHECK_FOREIGNS
Analysis:
========
A foreign key constraint cannot reference a secondary index defined
on a generated virtual column. While adding new index/drop existing
column, server internally drops the internal foreign key index and
it leads to choose the virtual secondary index as foreign key index.
But innodb doesn't allow foreign key constraint reference to
secondary virtual index.

Fix:
===
Allow foreign key constraint refer to secondary index defined on
a generated virutal column.

Reviewed-by: Jimmy Yang<jimmy.yang@oracle.com>
RB: 13586
2017-04-26 23:03:27 +03:00
Marko Mäkelä
93078c9cb1 Bug#24444831 MY_ERROR(ER_INNODB_ONLINE_LOG_TOO_BIG) CALLED WITH INVALID INDEX NAME
This bug was introduced in MySQL 5.6.8 with WL#6255.
When an error occurs while rebuilding a table that only has a
hidden GEN_CLUST_INDEX inside InnoDB, ha_alter_info->key_info_buffer
would be invalid and should not be dereferenced.

get_error_key_name(): Get the name of an erroneous key.
Avoid dereferencing ha_alter_info->key_info_buffer when no keys
exist in the SQL layer.

ha_innobase::inplace_alter_table(),
ha_innobase::commit_try_rebuild(): Invoke get_error_key_name()
for reporting ER_INNODB_ONLINE_LOG_TOO_BIG or ER_INDEX_CORRUPT.

RB: 13834
Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>
2017-04-26 23:03:27 +03:00
Marko Mäkelä
9a848ee182 Fix the -DWITH_INNODB_AHI=OFF build 2017-04-26 23:02:53 +03:00
Jan Lindström
765a43605a MDEV-12253: Buffer pool blocks are accessed after they have been freed
Problem was that bpage was referenced after it was already freed
from LRU. Fixed by adding a new variable encrypted that is
passed down to buf_page_check_corrupt() and used in
buf_page_get_gen() to stop processing page read.

This patch should also address following test failures and
bugs:

MDEV-12419: IMPORT should not look up tablespace in
PageConverter::validate(). This is now removed.

MDEV-10099: encryption.innodb_onlinealter_encryption fails
sporadically in buildbot

MDEV-11420: encryption.innodb_encryption-page-compression
failed in buildbot

MDEV-11222: encryption.encrypt_and_grep failed in buildbot on P8

Removed dict_table_t::is_encrypted and dict_table_t::ibd_file_missing
and replaced these with dict_table_t::file_unreadable. Table
ibd file is missing if fil_get_space(space_id) returns NULL
and encrypted if not. Removed dict_table_t::is_corrupted field.

Ported FilSpace class from 10.2 and using that on buf_page_check_corrupt(),
buf_page_decrypt_after_read(), buf_page_encrypt_before_write(),
buf_dblwr_process(), buf_read_page(), dict_stats_save_defrag_stats().

Added test cases when enrypted page could be read while doing
redo log crash recovery. Also added test case for row compressed
blobs.

btr_cur_open_at_index_side_func(),
btr_cur_open_at_rnd_pos_func(): Avoid referencing block that is
NULL.

buf_page_get_zip(): Issue error if page read fails.

buf_page_get_gen(): Use dberr_t for error detection and
do not reference bpage after we hare freed it.

buf_mark_space_corrupt(): remove bpage from LRU also when
it is encrypted.

buf_page_check_corrupt(): @return DB_SUCCESS if page has
been read and is not corrupted,
DB_PAGE_CORRUPTED if page based on checksum check is corrupted,
DB_DECRYPTION_FAILED if page post encryption checksum matches but
after decryption normal page checksum does not match. In read
case only DB_SUCCESS is possible.

buf_page_io_complete(): use dberr_t for error handling.

buf_flush_write_block_low(),
buf_read_ahead_random(),
buf_read_page_async(),
buf_read_ahead_linear(),
buf_read_ibuf_merge_pages(),
buf_read_recv_pages(),
fil_aio_wait():
        Issue error if page read fails.

btr_pcur_move_to_next_page(): Do not reference page if it is
NULL.

Introduced dict_table_t::is_readable() and dict_index_t::is_readable()
that will return true if tablespace exists and pages read from
tablespace are not corrupted or page decryption failed.
Removed buf_page_t::key_version. After page decryption the
key version is not removed from page frame. For unencrypted
pages, old key_version is removed at buf_page_encrypt_before_write()

dict_stats_update_transient_for_index(),
dict_stats_update_transient()
        Do not continue if table decryption failed or table
        is corrupted.

dict0stats.cc: Introduced a dict_stats_report_error function
to avoid code duplication.

fil_parse_write_crypt_data():
        Check that key read from redo log entry is found from
        encryption plugin and if it is not, refuse to start.

PageConverter::validate(): Removed access to fil_space_t as
tablespace is not available during import.

Fixed error code on innodb.innodb test.

Merged test cased innodb-bad-key-change5 and innodb-bad-key-shutdown
to innodb-bad-key-change2.  Removed innodb-bad-key-change5 test.
Decreased unnecessary complexity on some long lasting tests.

Removed fil_inc_pending_ops(), fil_decr_pending_ops(),
fil_get_first_space(), fil_get_next_space(),
fil_get_first_space_safe(), fil_get_next_space_safe()
functions.

fil_space_verify_crypt_checksum(): Fixed bug found using ASAN
where FIL_PAGE_END_LSN_OLD_CHECKSUM field was incorrectly
accessed from row compressed tables. Fixed out of page frame
bug for row compressed tables in
fil_space_verify_crypt_checksum() found using ASAN. Incorrect
function was called for compressed table.

Added new tests for discard, rename table and drop (we should allow them
even when page decryption fails). Alter table rename is not allowed.
Added test for restart with innodb-force-recovery=1 when page read on
redo-recovery cant be decrypted. Added test for corrupted table where
both page data and FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION is corrupted.

Adjusted the test case innodb_bug14147491 so that it does not anymore
expect crash. Instead table is just mostly not usable.

fil0fil.h: fil_space_acquire_low is not visible function
and fil_space_acquire and fil_space_acquire_silent are
inline functions. FilSpace class uses fil_space_acquire_low
directly.

recv_apply_hashed_log_recs() does not return anything.
2017-04-26 15:19:16 +03:00