Analysis: Problem is that both encrypted tables and compressed tables use
FIL header offset FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION to store
required metadata. Furhermore, for only compressed tables currently
code skips compression.
Fixes:
- Only encrypted pages store key_version to FIL header offset FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION,
no need to fix
- Only compressed pages store compression algorithm to FIL header offset FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION,
no need to fix as they have different page type FIL_PAGE_PAGE_COMPRESSED
- Compressed and encrypted pages now use a new page type FIL_PAGE_PAGE_COMPRESSED_ENCRYPTED and
key_version is stored on FIL header offset FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION and compression
method is stored after FIL header similar way as compressed size, so that first
FIL_PAGE_COMPRESSED_SIZE is stored followed by FIL_PAGE_COMPRESSION_METHOD
- Fix buf_page_encrypt_before_write function to really compress pages if compression is enabled
- Fix buf_page_decrypt_after_read function to really decompress pages if compression is used
- Small style fixes
Analysis: Problem was that we did try to read from tablespace
that was being dropped.
Fixed by introducing a new function to find a tablespace only
if it is not being dropped currently and adding this check
before trying to read pages from tablespace.
Analysis: At fil_spage_get_space there is small change that space
is found from tablespace list but we have not yet created node
for it (and added it to space->chain) and as we hold fil_system
mutex here fil_node_create can't continue.
Fixed by allowing UT_LIST_GET_LEN(space->chain) == 0|| 1 and
introducint two new functions that access filespace list
and before returning space check that node is also created.
Step 2:
-- Introduce temporal memory array to buffer pool where to allocate
temporary memory for encryption/compression
-- Rename PAGE_ENCRYPTION -> ENCRYPTION
-- Rename PAGE_ENCRYPTION_KEY -> ENCRYPTION_KEY
-- Rename innodb_default_page_encryption_key -> innodb_default_encryption_key
-- Allow enable/disable encryption for tables by changing
ENCRYPTION to enum having values DEFAULT, ON, OFF
-- In create table store crypt_data if ENCRYPTION is ON or OFF
-- Do not crypt tablespaces having ENCRYPTION=OFF
-- Store encryption mode to crypt_data and redo-log
Step 1:
-- Remove page encryption from dictionary (per table
encryption will be handled by storing crypt_data to page 0)
-- Remove encryption/compression from os0file and all functions
before that (compression will be added to buf0buf.cc)
-- Use same CRYPT_SCHEME_1 for all encryption methods
-- Do some code cleanups to confort InnoDB coding style
This patch ports the work that facebook has performed
to make innochecksum handle compressed tables.
the basic idea is to use actual innodb-code to perform
checksum verification rather than duplicating in innochecksum.cc.
to make this work, innodb code has been annotated with
lots of #ifndef UNIV_INNOCHECKSUM so that it can be
compiled outside of storage/innobase.
A new testcase is also added that verifies that innochecksum
works on compressed/non-compressed tables.
Merged from commit fabc79d2ea976c4ff5b79bfe913e6bc03ef69d42
from https://code.google.com/p/google-mysql/
The actual steps to produce this patch are:
take innochecksum from 5.6.14
apply changes in innodb from facebook patches needed to make innochecksum compile
apply changes in innochecksum from facebook patches
add handcrafted testcase
The referenced facebook patches used are:
91e25120e7847fe76ea51135628a5a4dbf7c240c
Merge Facebook commit f981a51a47519b0ba527917887f8adc6df9ae147
authored by Steaphan Greene from https://github.com/facebook/mysql-5.6.
This just moves some structure definitions from inside a
single .cc file to a shared .h file, with a few tweaks to
allow these structures to be shared.
On its own, it should have no actual effect. This is needed later.
Merged lp:maria/maria-10.0-galera up to revision 3880.
Added a new functions to handler API to forcefully abort_transaction,
producing fake_trx_id, get_checkpoint and set_checkpoint for XA. These
were added for future possiblity to add more storage engines that
could use galera replication.
buf0buf.cc line 2642.
Analysis: innodb_compression_algorithm is a global variable and
can change while we are building page compressed page. This could
lead page corruption.
Fix: Cache innodb_compression_algorithm on local variable before
doing any compression or page formating to avoid concurrent
change. Improved page verification on debug builds.
Analysis: InnoDB writes also files that do not contain FIL-header.
This could lead incorrect analysis on os_fil_read_func function
when it tries to see is page page compressed based on FIL_PAGE_TYPE
field on FIL-header. With bad luck uncompressed page that does
not contain FIL-headed, the byte on FIL_PAGE_TYPE position could
indicate that page is page comrpessed.
Fix: Upper layer must indicate is file space page compressed
or not. If this is not yet known, we need to read the FIL-header
and find it out. Files that we know that are not page compressed
we can always just provide FALSE.
Added option to disable multi-threaded flush with innodb_use_mtflush = 0
option, by default multi-threaded flush is used.
Updated innochecksum tool, still it does not support new checksums.
Update InnoDB to 5.6.14
Apply MySQL-5.6 hack for MySQL Bug#16434374
Move Aria-only HA_RTREE_INDEX from my_base.h to maria_def.h (breaks an assert in InnoDB)
Fix InnoDB memory leak
written and rest of the page is trimmed. In following writes
there is no need to trim again if write_size only increases
because rest of the page is already trimmed. If actual write
size decreases we need to trim again. Need to research if this
can happen frequently enough to make any effect.
Backporting the WL#5716, "Information schema table for InnoDB
buffer pool information". Backporting revisions 2876.244.113,
2876.244.102 from mysql-trunk.
rb://1177 approved by Jimmy Yang.
rb://942
approved by: Marko Makela
We don't need to scan LRU for dropping AHI entries when DROPing a table.
AHI entries are already removed when we free up extents for the btree.
This bug ensures that a live downgrade from an InnoDB 5.6 with WL5756 and
a database created with innodb-page-size=8k or 4k will make a version 5.5
installation politely refuse to start. Instead of crashing or giving some
indication about a corrupted database, it will indicate the page size
difference.
This patch takes only that part of the Wl5756 patch that is needed to
protect against opening a tablespace that is stamped with a different
page size.
It also contains the change in dict_index_find_on_id_low() just in case
a database with another page size is created by recompiling a pre-WL5756
InnoDB.
This patch doesn't get rid of the need to acquire the dict_sys->mutex but
reduces the need to keep the mutex locked for the duration of the query
to fsp_get_available_space_in_free_extents() from ha_innobase::info().
rb://390.
------------------------------------------------------------
revno: 3479
revision-id: marko.makela@oracle.com-20100524110439-fazi70rlmt07tzd9
parent: vasil.dimov@oracle.com-20100520133157-42uk5q3pp0vsinac
committer: Marko Mäkelä <marko.makela@oracle.com>
branch nick: 5.1-innodb
timestamp: Mon 2010-05-24 14:04:39 +0300
message:
Bug#53578: assert on invalid page access, in fil_io()
Store the max_space_id in the data dictionary header in order to avoid
space_id reuse.
DICT_HDR_MIX_ID: Renamed to DICT_HDR_MAX_SPACE_ID, DICT_HDR_MIX_ID_LOW.
dict_hdr_get_new_id(): Return table_id, index_id, space_id or a subset of them.
fil_system_t: Add ibool space_id_reuse_warned.
fil_create_new_single_table_tablespace(): Get the space_id from the caller.
fil_space_create(): Issue a warning if the fil_system->max_assigned_id
is exceeded.
fil_assign_new_space_id(): Return TRUE/FALSE and take a pointer to the
space_id as a parameter. Make the function public.
fil_init(): Initialize all fil_system fields by mem_zalloc(). Remove
explicit initializations of certain fields to 0 or NULL.
not address the printouts issue
Detailed revision comments:
r6310 | marko | 2009-12-15 15:23:54 +0200 (Tue, 15 Dec 2009) | 30 lines
branches/5.1: Merge r4922 from branches/zip.
This the fix for the first part of Bug #41609 from InnoDB Plugin to
the built-in InnoDB in MySQL 5.1. This allows InnoDB Hot Backup to
back up a database while the built-in InnoDB in MySQL 5.1 is creating
temporary tables. (This fix does not address the printouts about
missing .ibd files for temporary tables at InnoDB startup, which was
committed to branches/zip in r6252.)
rb://219 approved by Sunny Bains.
branches/zip: Distinguish temporary tables in MLOG_FILE_CREATE.
This addresses Mantis Issue #23 in InnoDB Hot Backup and some
of MySQL Bug #41609.
In MLOG_FILE_CREATE, we need to distinguish temporary tables, so that
InnoDB Hot Backup can work correctly. It turns out that we can do this
easily, by using a bit of the previously unused parameter for page number.
(The page number parameter of MLOG_FILE_CREATE has been written as 0
ever since MySQL 4.1, which introduced MLOG_FILE_CREATE.)
MLOG_FILE_FLAG_TEMP: A flag for indicating a temporary table in
the page number parameter of MLOG_FILE_ operations.
fil_op_write_log(): Add the parameter log_flags.
fil_op_log_parse_or_replay(): Add the parameter log_flags.
Do not replay MLOG_FILE_CREATE when MLOG_FILE_FLAG_TEMP is set in log_flags.
This only affects ibbackup --apply-log. InnoDB itself never replays file
operations.
layout as we always had in trees containing only the builtin
2) win\configure.js WITH_INNOBASE_STORAGE_ENGINE still works.
storage/innobase/CMakeLists.txt:
fix to new directory name (and like 5.1)
storage/innobase/Makefile.am:
fix to new directory name (and like 5.1)
storage/innobase/handler/ha_innodb.cc:
fix to new directory name (and like 5.1)
storage/innobase/plug.in:
fix to new directory name (and like 5.1)
bzr branch mysql-5.1-performance-version mysql-trunk # Summit
cd mysql-trunk
bzr merge mysql-5.1-innodb_plugin # which is 5.1 + Innodb plugin
bzr rm innobase # remove the builtin
Next step: build, test fixes.
Fixed BUG#19542 "InnoDB doesn't increase the Handler_read_prev couter".
Fixed BUG#19609 "Case sensitivity of innodb_data_file_path gives stupid error".
Fixed BUG#19727 "InnoDB crashed server and crashed tables are ot recoverable".
Also:
* Remove remnants of the obsolete concept of memoryfixing tables and indexes.
* Remove unused dict_table_LRU_trim().
* Remove unused 'trx' parameter from dict_table_get_on_id_low(),
dict_table_get(), dict_table_get_and_increment_handle_count().
* Add a normal linked list implementation.
* Add a work queue implementation.
* Add 'level' parameter to mutex_create() and rw_lock_create().
Remove mutex_set_level() and rw_lock_set_level().
* Rename SYNC_LEVEL_NONE to SYNC_LEVEL_VARYING.
* Add support for bound ids in InnoDB's parser.
* Define UNIV_BTR_DEBUG for enabling consistency checks of
FIL_PAGE_NEXT and FIL_PAGE_PREV when accessing sibling
pages of B-tree indexes.
btr_validate_level(): Check the validity of the doubly linked
list formed by FIL_PAGE_NEXT and FIL_PAGE_PREV.
* Adapt InnoDB to the new tablename to filename encoding in MySQL 5.1.
ut_print_name(), ut_print_name1(): Add parameter 'table_id' for
distinguishing names of tables from other identifiers.
New: innobase_convert_from_table_id(), innobase_convert_from_id(),
innobase_convert_from_filename(), innobase_get_charset.
dict_accept(), dict_scan_id(), dict_scan_col(), dict_scan_table_name(),
dict_skip_word(), dict_create_foreign_constraints_low(): Add
parameter 'cs' so that isspace() can be replaced with my_isspace(),
whose operation depends on the connection character set.
dict_scan_id(): Convert identifier to UTF-8.
dict_str_starts_with_keyword(): New extern function, to replace
dict_accept() in row_search_for_mysql().
mysql_get_identifier_quote_char(): Replaced with innobase_print_identifier().
ha_innobase::create(): Remove the thd->convert_strin() call. Pass the
statement to InnoDB in the connection character set and let InnoDB
convert the identifier to UTF-8.
* Add max_row_size to dict_table_t.
* btr0cur.c
btr_copy_externally_stored_field(): Only set the 'offset' variable
when needed.
* buf0buf.c
buf_page_io_complete(): Write to the error log if the page number or
the space id o the disk do not match those in memory. Also write to
the error log if a page was read from the doublewrite buffer. The
doublewrite buffer should be only read by the lower-level function
fil_io() at database startup.
* dict0dict.c
dict_scan_table_name(): Remove fallback to differently encoded name
when the table is not found. The encoding is handled at a higher level.
* ha_innodb.cc
Increment statistic counter in ha_innobase::index_prev() (bug 19542).
Add innobase_convert_string wrapper function and a new file
ha_prototypes.h.
innobase_print_identifier(): Remove TODO comment before calling
get_quote_char_for_identifier(). That function apparently assumes
the identifier to be encoded in UTF-8.
* ibuf0ibuf.c|h
ibuf_count_get(), ibuf_counts[], ibuf_count_inited(): Define these
only #ifdef UNIV_IBUF_DEBUG. Previously, when compiled without
UNIV_IBUF_DEBUG, invoking ibuf_count_get() would crash InnoDB.
The function is only being called #ifdef UNIV_IBUF_DEBUG.
* innodb.result
Adjust the results for changes in the foreign key error messages.
* mem0mem.c|h
New: mem_heap_dup(), mem_heap_printf(), mem_heap_cat().
* os0file.c
Check the page trailers also after writing to disk. This improves
chances of diagnosing bug 18886.
os_file_check_page_trailers(): New function for checking that the
two copies of the LSN stamped on the page match.
os_aio_simulated_handle(): Call os_file_check_page_trailers()
before and after os_file_write().
* row0mysql.c
Move trx_commit_for_mysql(trx) calls before calls to
row_mysql_unlock_data_dictionary(trx) (bug 19727).
* row0sel.c
row_fetch_print(): Handle SQL NULL values without crashing.
row_sel_store_mysql_rec(): Remove useless call to rec_get_nth_field
when handling an externally stored column.
Fetch externally stored fields when using InnoDB's internal SQL
parser.
Optimize BLOB selects by using prebuilt->blob_heap directly instead
of first reading BLOB data to a temporary heap and then copying it
to prebuilt->blob_heap.
* srv0srv.c
srv_master_thread(): Remove unreachable code.
* srv0start.c
srv_parse_data_file_paths_and_sizes(): Accept lower-case 'm' and
'g' as abbreviations of megabyte and gigabyte (bug 19609).
srv_parse_megabytes(): New fuction.
* ut0dbg.c|h
Implement InnoDB assertions (ut_a and ut_error) with abort() when
the code is compiled with GCC 3 or later on other platforms than
Windows or Netware. Also disable the variable ut_dbg_stop_threads
and the function ut_dbg_stop_thread() i this case, unless
UNIV_SYC_DEBUG is defined. This should allow the compiler to
generate more compact code for assertions.
* ut0list.c|h
Add ib_list_create_heap().
mysql-test/r/innodb.result:
Applied innodb-5.1-ss594 snapshot.
mysql-test/t/innodb.test:
Copy the comment from the beginning of the file
to the end because MySQL developers continue
adding test cases to this file.
sql/ha_innodb.cc:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/CMakeLists.txt:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/Makefile.am:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/btr/btr0btr.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/btr/btr0cur.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/btr/btr0pcur.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/btr/btr0sea.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/buf/buf0buf.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/buf/buf0flu.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/data/data0type.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/dict/dict0crea.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/dict/dict0dict.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/dict/dict0load.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/dict/dict0mem.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/fil/fil0fil.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/ha/hash0hash.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/ibuf/ibuf0ibuf.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/data0type.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/dict0dict.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/dict0dict.ic:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/dict0mem.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/fil0fil.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/fsp0fsp.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/ibuf0ibuf.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/mem0mem.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/pars0pars.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/pars0sym.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/pars0types.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/rem0rec.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/row0purge.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/row0undo.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/srv0srv.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/sync0rw.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/sync0sync.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/univ.i:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/ut0dbg.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/ut0ut.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/lock/lock0lock.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/log/log0log.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/log/log0recv.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/mem/mem0dbg.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/mem/mem0mem.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/mem/mem0pool.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/os/os0file.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/os/os0thread.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/pars/lexyy.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/pars/pars0lex.l:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/pars/pars0pars.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/pars/pars0sym.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/row/row0ins.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/row/row0mysql.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/row/row0purge.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/row/row0row.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/row/row0sel.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/row/row0upd.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/srv/srv0srv.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/srv/srv0start.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/sync/sync0arr.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/sync/sync0rw.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/sync/sync0sync.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/thr/thr0loc.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/trx/trx0purge.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/trx/trx0rec.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/trx/trx0roll.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/trx/trx0rseg.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/trx/trx0sys.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/trx/trx0trx.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/ut/Makefile.am:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/ut/ut0dbg.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/ut/ut0ut.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/ha_prototypes.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/ut0list.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/ut0list.ic:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/include/ut0wqueue.h:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/ut/ut0list.c:
Applied innodb-5.1-ss594 snapshot.
storage/innobase/ut/ut0wqueue.c:
Applied innodb-5.1-ss594 snapshot.
