Commit graph

285 commits

Author SHA1 Message Date
Marko Mäkelä
8451e09073 MDEV-11556 InnoDB redo log apply fails to adjust data file sizes
fil_space_t::recv_size: New member: recovered tablespace size in pages;
0 if no size change was read from the redo log,
or if the size change was implemented.

fil_space_set_recv_size(): New function for setting space->recv_size.

innodb_data_file_size_debug: A debug parameter for setting the system
tablespace size in recovery even when the redo log does not contain
any size changes. It is hard to write a small test case that would
cause the system tablespace to be extended at the critical moment.

recv_parse_log_rec(): Note those tablespaces whose size is being changed
by the redo log, by invoking fil_space_set_recv_size().

innobase_init(): Correct an error message, and do not require a larger
innodb_buffer_pool_size when starting up with a smaller innodb_page_size.

innobase_start_or_create_for_mysql(): Allow startup with any initial
size of the ibdata1 file if the autoextend attribute is set. Require
the minimum size of fixed-size system tablespaces to be 640 pages,
not 10 megabytes. Implement innodb_data_file_size_debug.

open_or_create_data_files(): Round the system tablespace size down
to pages, not to full megabytes, (Our test truncates the system
tablespace to more than 800 pages with innodb_page_size=4k.
InnoDB should not imagine that it was truncated to 768 pages
and then overwrite good pages in the tablespace.)

fil_flush_low(): Refactored from fil_flush().

fil_space_extend_must_retry(): Refactored from
fil_extend_space_to_desired_size().

fil_mutex_enter_and_prepare_for_io(): Extend the tablespace if
fil_space_set_recv_size() was called.

The test case has been successfully run with all the
innodb_page_size values 4k, 8k, 16k, 32k, 64k.
2016-12-30 09:52:24 +02:00
Marko Mäkelä
d50cf42bc0 MDEV-9282 Debian: the Lintian complains about "shlib-calls-exit" in ha_innodb.so
Replace all exit() calls in InnoDB with abort() [possibly via ut_a()].
Calling exit() in a multi-threaded program is problematic also for
the reason that other threads could see corrupted data structures
while some data structures are being cleaned up by atexit() handlers
or similar.

In the long term, all these calls should be replaced with something
that returns an error all the way up the call stack.
2016-12-28 15:54:24 +02:00
Marko Mäkelä
7e02fd1f71 MDEV-11630 Call mutex_free() before freeing the mutex list
Make some global fil_crypt_ variables static.

fil_close(): Call mutex_free(&fil_system->mutex) also in InnoDB, not
only in XtraDB. In InnoDB, sync_close() was called before fil_close().

innobase_shutdown_for_mysql(): Call fil_close() before sync_close(),
similar to XtraDB shutdown.

fil_space_crypt_cleanup(): Call mutex_free() to pair with
fil_space_crypt_init().

fil_crypt_threads_cleanup(): Call mutex_free() to pair with
fil_crypt_threads_init().
2016-12-22 14:33:58 +02:00
Jan Lindström
72cc73cea2 MDEV-10368: get_latest_version() called too often
Reduce the number of calls to encryption_get_key_get_latest_version
when doing key rotation with two different methods:

(1) We need to fetch key information when tablespace not yet
have a encryption information, invalid keys are handled now
differently (see below). There was extra call to detect
if key_id is not found on key rotation.

(2) If key_id is not found from encryption plugin, do not
try fetching new key_version for it as it will fail anyway.
We store return value from encryption_get_key_get_latest_version
call and if it returns ENCRYPTION_KEY_VERSION_INVALID there
is no need to call it again.
2016-12-13 11:51:33 +02:00
Jan Lindström
2bedc3978b MDEV-9931: InnoDB reads first page of every .ibd file at startup
Analysis: By design InnoDB was reading first page of every .ibd file
at startup to find out is tablespace encrypted or not. This is
because tablespace could have been encrypted always, not
encrypted newer or encrypted based on configuration and this
information can be find realible only from first page of .ibd file.

Fix: Do not read first page of every .ibd file at startup. Instead
whenever tablespace is first time accedded we will read the first
page to find necessary information about tablespace encryption
status.

TODO: Add support for SYS_TABLEOPTIONS where all table options
encryption information included will be stored.
2016-09-22 16:38:24 +03:00
Sergei Golubchik
6b1863b830 Merge branch '10.0' into 10.1 2016-08-25 12:40:09 +02:00
Sergei Golubchik
57fbc603bf Merge branch 'merge/merge-innodb-5.6' into 10.0
5.6.32
2016-08-10 19:43:37 +02:00
Sergei Golubchik
b4f97a1499 5.6.32 2016-08-10 19:23:00 +02:00
Sergei Golubchik
3361aee591 Merge branch '10.0' into 10.1 2016-06-28 22:01:55 +02:00
Sergei Golubchik
a79d46c3a4 Merge branch 'merge-innodb-5.6' into 10.0 2016-06-21 14:58:19 +02:00
Sergei Golubchik
720e04ff67 5.6.31 2016-06-21 14:21:03 +02:00
Sergei Golubchik
a5679af1b1 Merge branch '10.0' into 10.1 2016-02-23 21:35:05 +01:00
Sergei Golubchik
c8fcaf8aec Merge branch 'merge-innodb-5.6' into 10.0 2016-02-16 18:32:59 +01:00
Sergei Golubchik
d520d35380 Revert "MDEV-8696: Adding indexes on empty table is slow with large innodb_sort_buffer_size."
This reverts commit 13884cf206.

Fixed in 5.6 (merged in the next commit)
2016-02-16 12:53:24 +01:00
Sergei Golubchik
220e70fadc 5.6.29 2016-02-16 12:07:18 +01:00
Jan Lindström
111acb721f MDEV-9359: encryption.create_or_replace fails sporadically in buildbot: failing assertion: mutex->magic_n == MUTEX_MAGIC_N
Make sure that encryption threads mutex is initialized before starting
encryption threads.
2016-01-05 18:50:54 +02:00
Sergei Golubchik
dfb58a3782 innodb/xtradb: init scrub mutex even in read-only mode
because it's locked for innodb part of SHOW STATUS,
which can happen in read-only mode too.
2015-12-21 21:25:01 +01:00
Sergei Golubchik
a2bcee626d Merge branch '10.0' into 10.1 2015-12-21 21:24:22 +01:00
Jan Lindström
953d5680a3 Merge pull request #133 from grooverdan/power-crc32
MDEV-9288: Use POWER8 accelerated crc32
2015-12-16 09:34:24 +02:00
Jan Lindström
b63bf7368b MDEV-8923: port innodb_buffer_pool_dump_pct from MySQL
Backport pull request #125 from grooverdan/MDEV-8923_innodb_buffer_pool_dump_pct to 10.0

WL#6504 InnoDB buffer pool dump/load enchantments

This patch consists of two parts:

    1. Dump only the hottest N% of the buffer pool(s)
    2. Prevent hogging the server duing BP load

From MySQL - commit b409342c43ce2edb68807100a77001367c7e6b8e

Add testcases for innodb_buffer_pool_dump_pct_basic.

Part of the code authored by Daniel Black
2015-12-15 09:35:22 +02:00
Daniel Axtens
2538c7cf89 Use POWER8 accelerated crc32
- Make accelerated checksum available to InnoDB and XtraDB.
 - Fall back to slice-by-eight if not available. The mode used is printed on startup.
 - Will only build on POWER systems at the moment until CMakeLists are modified
   to only add the crc32_power8/ files when building on POWER.

running MySQL-5.7 unittest/gunit/innodb/ut0crc32-t

Before:

1..2
Using software crc32 implementation, CPU is little-endian
ok 1
Using software crc32 implementation, CPU is little-endian
    normal CRC32: real    0.148006 sec
    normal CRC32: user    0.148000 sec
    normal CRC32: sys     0.000000 sec
big endian CRC32: real    0.144293 sec
big endian CRC32: user    0.144000 sec
big endian CRC32: sys     0.000000 sec
ok 2

After:

1..2
Using POWER8 crc32 implementation, CPU is little-endian
ok 1
Using POWER8 crc32 implementation, CPU is little-endian
    normal CRC32: real    0.008097 sec
    normal CRC32: user    0.008000 sec
    normal CRC32: sys     0.000000 sec
big endian CRC32: real    0.147043 sec
big endian CRC32: user    0.144000 sec
big endian CRC32: sys     0.000000 sec
ok 2

Author CRC32 ASM code: Anton Blanchard <anton@au.ibm.com>
ref: https://github.com/antonblanchard/crc32-vpmsum

Signed-off-by: Daniel Black <daniel.black@au.ibm.com>
2015-12-15 15:11:17 +11:00
Sergei Golubchik
5b3c100788 Merge branch 'merge/merge-innodb-5.6' into 10.0 2015-12-13 10:18:42 +01:00
Sergei Golubchik
e9eaaa4b4a 5.6.28 2015-12-13 10:11:49 +01:00
Vasil Dimov
a816df7009 MDEV-8923 Port innodb_buffer_pool_dump_pct
WL#6504 InnoDB buffer pool dump/load enchantments

This patch consists of two parts:

1. Dump only the hottest N% of the buffer pool(s)
2. Prevent hogging the server duing BP load

From MySQL - commit b409342c43ce2edb68807100a77001367c7e6b8e
2015-11-29 18:08:42 +11:00
Sergei Golubchik
beded7d9c9 Merge branch '10.0' into 10.1 2015-11-19 15:52:14 +01:00
Jan Lindström
13884cf206 MDEV-8696: Adding indexes on empty table is slow with large innodb_sort_buffer_size.
Analysis: Current implementation will write and read at least one block
(sort_buffer_size bytes) from disk / index even if that block does not
contain any records.

Fix: Avoid writing / reading empty blocks to temporary files (disk).
2015-10-27 13:00:15 +02:00
Sergei Golubchik
dfb74dea30 Merge branch '10.0' into 10.1 2015-10-12 00:37:58 +02:00
Sergei Golubchik
04af573d65 Merge branch 'merge-innodb-5.6' into 10.0 2015-10-09 17:47:30 +02:00
Sergei Golubchik
86ff4da14d 5.6.27 2015-10-09 17:21:46 +02:00
Eric Herman
b7abdcf5c7 Add lru evict count for the idle loop (innobase)
port of work by dveeden:
555ca34958
2015-08-26 11:49:21 +02:00
Eric Herman
4f4373fe31 Add eviction count for table cache lru cleanup (innobase)
This is a port of dveeden's work:

e9d00e859e

(Mostly changes in linenumbers, a bit of whitespace fix-up.)
2015-08-26 10:24:03 +02:00
Nirbhay Choubey
dced5146bd Merge branch '10.0-galera' into 10.1 2015-07-14 16:05:29 -04:00
Sergei Golubchik
658992699b Merge tag 'mariadb-10.0.20' into 10.1 2015-06-27 20:35:26 +02:00
Sergei Golubchik
fe7e334f3e cleanup: remove unused function argument 2015-06-27 09:40:57 +02:00
Jan Lindström
eb2c170bb0 MDEV-8303; Dumping buffer pool noisy in the logs.
Added new dynamic configuration variable innodb_buf_dump_status_frequency
to configure how often buffer pool dump status is printed in the logs.
A number between [0, 100] that tells how oftern buffer pool dump status
in percentages should be printed. E.g. 10 means that buffer pool dump
status is printed when every 10% of number of buffer pool pages are
dumped. Default is 0 (only start and end status is printed).
2015-06-18 12:20:21 +03:00
Sergei Golubchik
985e430c0f after-merge fixes
in innobase: compilation error on windows
other changes: perfschema merge followup
2015-06-16 23:55:56 +02:00
Sergei Golubchik
a65162a396 Merge branch 'merge-innodb-5.6' into 10.0 2015-06-16 11:08:23 +02:00
Sergei Golubchik
139ba26dba 5.6.25 2015-06-16 10:57:05 +02:00
Jan Lindström
59815a268b MDEV-7484: Log Time not consistent with InnoDB errors nor with MySQL error log time format 2015-05-30 20:45:17 +03:00
Jan Lindström
f8cacd03a7 MDEV-8143: InnoDB: Database page corruption on disk or a failed file read
Analysis: Problem was that we did create crypt data for encrypted table but
this new crypt data was not written to page 0. Instead a default crypt data
was written to page 0 at table creation.

Fixed by explicitly writing new crypt data to page 0 after successfull
table creation.
2015-05-14 11:32:24 +03:00
Jan Lindström
d259376fd8 MDEV-8041: InnoDB redo log encryption
Merged new version of InnoDB/XtraDB redo log encryption from Google
provided by Jonas Oreland.
2015-05-09 11:13:00 +03:00
Sergei Golubchik
6d06fbbd1d move to storage/innobase 2015-05-04 19:17:21 +02:00
Sergei Golubchik
ea764f5f52 s/innodb_default_encryption_key/innodb_default_encryption_key_id/
and make it a session variable, not global, as any decent
default sysvar for a table attribute should be
2015-04-10 02:40:10 +02:00
Sergei Golubchik
0a9052f591 Store the key id in the tablespace and read it back 2015-04-09 19:36:12 +02:00
Jan Lindström
af768c2f22 InnoDB/XtraDB Encryption code cleanup
Step 4:
-- Review fixes
-- Rename pages_page_* to pages_*
-- Remove unnecessary code
2015-04-07 23:44:57 +02:00
Jan Lindström
0ba9fa35bc InnoDB/XtraDB Encryption cleanup
Step 2:

-- Introduce temporal memory array to buffer pool where to allocate
temporary memory for encryption/compression
-- Rename PAGE_ENCRYPTION -> ENCRYPTION
-- Rename PAGE_ENCRYPTION_KEY -> ENCRYPTION_KEY
-- Rename innodb_default_page_encryption_key -> innodb_default_encryption_key
-- Allow enable/disable encryption for tables by changing
 ENCRYPTION to enum having values DEFAULT, ON, OFF
-- In create table store crypt_data if ENCRYPTION is ON or OFF
-- Do not crypt tablespaces having ENCRYPTION=OFF
-- Store encryption mode to crypt_data and redo-log
2015-04-07 23:44:56 +02:00
Jan Lindström
b4a4d82396 InnoDB/XtraDB Encryption cleanup.
Step 1:
-- Remove page encryption from dictionary (per table
encryption will be handled by storing crypt_data to page 0)
-- Remove encryption/compression from os0file and all functions
before that (compression will be added to buf0buf.cc)
-- Use same CRYPT_SCHEME_1 for all encryption methods
-- Do some code cleanups to confort InnoDB coding style
2015-04-07 23:44:56 +02:00
Sergei Golubchik
2db62f686e Merge branch '10.0' into 10.1 2015-03-07 13:21:02 +01:00
Jan Lindström
1cc7befc14 MDEV-7109: Add support for INFORMATION_SCHEMA.INNODB_SEMAPHORE_WAITS
MDEV-7399: Add support for INFORMATION_SCHEMA.INNODB_MUTEXES
    MDEV-7618: Improve semaphore instrumentation

    Introduced two new information schema tables to monitor mutex waits
    and semaphore waits. Added a new configuration variable
    innodb_intrument_semaphores to add thread_id, file name and
    line of current holder of mutex/rw_lock.
2015-02-21 21:45:16 +02:00
Sergei Golubchik
6b05688f6d innodb 5.6.23 2015-02-18 17:59:21 +01:00