mariadb/storage/maria/ma_key_recover.h

105 lines
4 KiB
C
Raw Normal View History

First part of redo/undo for key pages Added key_nr to st_maria_keydef for faster keyinfo->keynr conversion For transactional tables, shift record number in keys up with 1 bit to have place to indicate if transid follows Checksum for MyISAM now ignores NULL and not used part of VARCHAR Renamed some variables that caused shadow compiler warnings Moved extra() call when waiting for tables to not be used to after tables are removed from cache. Fixed crashing bugs when using Maria TEMPORARY tables with TRUNCATE. Removed 'hack' code in sql directory to go around this bug. pagecache_unlock_by_ulink() now has extra argument to say if page was changed. Give error message if we fail to open control file Mark page cache variables as not flushable include/maria.h: Made min page cache larger (needed for pinning key page) Added key_nr to st_maria_keydef for faster keyinfo->keynr conversion Added write_comp_flag to move some runtime code to maria_open() include/my_base.h: Added new error message to be used when handler initialization failed include/my_global.h: Renamed dummy to swap_dummy to avoid conflicts with local 'dummy' variables include/my_handler.h: Added const to some parameters mysys/array.c: More DBUG mysys/my_error.c: Fixed indentation mysys/my_handler.c: Added const to some parameters Added missing error messages sql/field.h: Renamed variables to avoid variable shadowing sql/handler.h: Renamed parameter to avoid variable name conflict sql/item.h: Renamed variables to avoid variable shadowing sql/log_event_old.h: Renamed variables to avoid variable shadowing sql/set_var.h: Renamed variables to avoid variable shadowing sql/sql_delete.cc: Removed maria hack for temporary tables Fixed indentation sql/sql_table.cc: Moved extra() call when waiting for tables to not be used to after tables are removed from cache. This was needed to ensure we don't do a PREPARE_FOR_DROP or similar call while the table is still in use. sql/table.cc: Copy page_checksum from share Removed Maria hack storage/maria/Makefile.am: Added new files storage/maria/ha_maria.cc: Renamed records -> record_count and info -> create_info to avoid variable name conflicts Mark page cache variables as not flushable storage/maria/ma_blockrec.c: Moved _ma_unpin_all_pages() to ma_key_recover.c Moved init of info->pinned_pages to ma_open.c Moved _ma_finalize_row() to maria_key_recover.h Renamed some variables to avoid variable name conflicts Mark page_link.changed for blocks we change directly Simplify handling of undo link when writing LOGREC_UNDO_ROW_INSERT (old code crashed when having redo for index) storage/maria/ma_blockrec.h: Removed extra empty line storage/maria/ma_checkpoint.c: Remove not needed trnman.h storage/maria/ma_close.c: Free pinned pages (which are now always allocated) storage/maria/ma_control_file.c: Give error message if we fail to open control file storage/maria/ma_delete.c: Changes for redo logging (first part, logging of underflow not yet done) - Log undo-key-delete - Log delete of key - Updated arguments to _ma_fetch_keypage(), _ma_dispose(), _ma_write_keypage(), _ma_insert() - Added new arguments to some functions to be able to write redo information - Mark key pages as changed when we write with PAGECACHE_LOCK_LEFT_WRITELOCKED Remove one not needed _ma_write_keypage() in d_search() when upper level will do the write anyway Changed 2 bmove_upp() to bmove() as this made code easer to understand More function comments Indentation fixes storage/maria/ma_ft_update.c: New arguments to _ma_write_keypage() storage/maria/ma_loghandler.c: Fixed some DBUG_PRINT messages Simplify code Added new log entrys for key page redo Renamed some variables to avoid variable name shadowing storage/maria/ma_loghandler.h: Moved some defines here Added define for storing key number on key pages Added new translog record types Added enum for type of operations in LOGREC_REDO_INDEX storage/maria/ma_open.c: Always allocate info.pinned_pages (we need now also for normal key page usage) Update keyinfo->key_nr Added virtual functions to convert record position o number to be stored on key pages Update keyinfo->write_comp_flag to value of search flag to be used when writing key storage/maria/ma_page.c: Added redo for key pages - Extended _ma_fetch_keypage() with type of lock to put on page and address to used MARIA_PINNED_PAGE - _ma_fetch_keypage() now pin's pages if needed - Extended _ma_write_keypage() with type of locks to be used - ma_dispose() now locks info->s->state.key_del from other threads - ma_dispose() writes redo log record - ma_new() locks info->s->state.key_del from other threads if it was used - ma_new() now pins read page Other things: - Removed some not needed arguments from _ma_new() and _ma_dispose) - Added some new variables to simplify code - If EXTRA_DEBUG is used, do crc on full page to catch not unitialized bytes storage/maria/ma_pagecache.h: Applied patch from Sanja to add extra argument to pagecache_unlock_by_ulink() to mark if page was changed Added some defines for pagecache priority levels that one can use storage/maria/ma_range.c: Added new arguments for call to _ma_fetch_keypage() storage/maria/ma_recovery.c: - Added hooks for new translog types: REDO_INDEX, REDO_INDEX_NEW_PAGE, REDO_INDEX_FREE_PAGE, UNDO_KEY_INSERT, UNDO_KEY_DELETE and UNDO_KEY_DELETE_WITH_ROOT. - Moved variable declarations to start of function (portability fixes) - Removed some not needed initializations - Set only relevant state changes for each redo/undo entry storage/maria/lockman.c: Removed end space storage/maria/ma_check.c: Removed end space storage/maria/ma_create.c: Removed end space storage/maria/ma_locking.c: Removed end space storage/maria/ma_packrec.c: Removed end space storage/maria/ma_pagecache.c: Removed end space storage/maria/ma_panic.c: Removed end space storage/maria/ma_rt_index.c: Added new arguments for call to _ma_fetch_keypage(), _ma_write_keypage(), _ma_dispose() and _ma_new() Fixed indentation storage/maria/ma_rt_key.c: Added new arguments for call to _ma_fetch_keypage() storage/maria/ma_rt_split.c: Added new arguments for call to _ma_new() Use new keypage header Added new arguments for call to _ma_write_keypage() storage/maria/ma_search.c: Updated comments & indentation Added new arguments for call to _ma_fetch_keypage() Made some variables and arguments const Added virtual functions for converting row position to number to be stored in key use MARIA_RECORD_POS of record position instead of my_off_t Record in MARIA_KEY_PARAM how page was changed one key insert (needed for REDO) storage/maria/ma_sort.c: Removed end space storage/maria/ma_statrec.c: Updated arguments for call to _ma_rec_pos() storage/maria/ma_test1.c: Fixed too small buffer to init_pagecache() Fixed bug when using insert_count and test_flag storage/maria/ma_test2.c: Use more resonable pagecache size Remove not used code Reset blob_length to fix wrong output message storage/maria/ma_test_all.sh: Fixed wrong test storage/maria/ma_write.c: Lots of new code to handle REDO of key pages No logic changes because of REDO code, mostly adding new arguments and adding new code for logging Added new arguments for calls to _ma_fetch_keypage(), _ma_write_keypage() and similar functions Move setting of comp_flag in ma_ck_wrte_btree() from runtime to maria_open() Zerofill new used pages for: - To remove possible sensitive data left in buffer - To get idenitical data on pages after running redo - Better compression of pages if archived storage/maria/maria_chk.c: Added information if table is crash safe storage/maria/maria_def.h: New virtual function to convert between record position on key and normal record position Aded mutex and extra variables to handle locking of share->state.key_del Moved some structure variables to get things more aligned Added extra arguments to MARIA_KEY_PARAM to be able to remember what was changed on key page on key insert Added argument to MARIA_PINNED_PAGE to indicate if page was changed Updated prototypes for functions Added some structures for signaling changes in REDO handling storage/maria/unittest/ma_pagecache_single.c: Updated arguments for changed function calls storage/myisam/mi_check.c: Made calc_check_checksum virtual storage/myisam/mi_checksum.c: Update checksums to ignore null columns storage/myisam/mi_create.c: Mark if table has null column (to know when we have to use mi_checksum()) storage/myisam/mi_open.c: Added virtual function for calculating checksum to be able to easily ignore NULL fields storage/myisam/mi_test2.c: Fixed bug storage/myisam/myisamdef.h: Added virtual function for calculating checksum during check table Removed ha_key_cmp() as this is in handler.h storage/maria/ma_key_recover.c: New BitKeeper file ``storage/maria/ma_key_recover.c'' storage/maria/ma_key_recover.h: New BitKeeper file ``storage/maria/ma_key_recover.h'' storage/maria/ma_key_redo.c: New BitKeeper file ``storage/maria/ma_key_redo.c''
2007-11-14 18:08:06 +01:00
/* Copyright (C) 2007 Michael Widenius
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; version 2 of the License.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */
/*
When we have finished the write/update/delete of a row, we have cleanups to
do. For now it is signalling to Checkpoint that all dirtied pages have
their rec_lsn set and page LSN set (_ma_unpin_all_pages() has been called),
and that bitmap pages are correct (_ma_bitmap_release_unused() has been
called).
*/
Fixes for redo/undo logging of key pages New extendable format for maria_log_control file Fixed some compiler warnings include/maria.h: Added maria_disable_logging() and maria_enable_logging() mysql-test/include/maria_verify_recovery.inc: Updated tests now when key redo/undo works mysql-test/r/maria-recovery.result: Updated tests now when key redo/undo works storage/maria/ma_blockrec.c: Use unified CLR code Added rec_lsn for full pages Moved clr write hook to ma_key_recover.c Changed REDO code to keep pages pinned until undo Mark page_link's as changed storage/maria/ma_blockrec.h: Moved write_hook_for_clr_end() to ma_key_recover.c storage/maria/ma_check.c: Changed key check code to use PAGECACHE_READ_UNKNOWN_PAGE Fixed wrong warning when checking files after maria_pack When unpacking files, we have to use new keypos_to_recpos method When doing repair, we can disregard index key file pages in page cache storage/maria/ma_commit.c: Added simple enable/disable logging functions (Needed for recovery) storage/maria/ma_control_file.c: Make maria control file extendable without having to make it incompatible for older versions storage/maria/ma_control_file.h: New error messages Added CONTROL_FILE_VERSION storage/maria/ma_delete.c: Added redo/undo for key pages change_length -> changed_length to make things similar More comments & more DBUG storage/maria/ma_key_recover.c: Unified CLR method Moved here write_hook_for_clr_end() and common keypage log functions Changed REDO to keep pages pinned until undo Changed UNDO code to change key_root under log mutex storage/maria/ma_key_recover.h: New structures and functions storage/maria/ma_loghandler.c: Include needed files storage/maria/ma_open.c: Change maria_open() to use pread() instead of read() storage/maria/ma_page.c: Fixed bug in key_del handling Clear pages if IDENTICAL_PAGES_AFTER_RECOVERY is defined storage/maria/ma_pagecache.c: Indentation and spelling fixes More DBUG Added helper function: pagecache_block_link_to_buffer() storage/maria/ma_pagecache.h: Added pagecache_block_link_to_buffer() storage/maria/ma_recovery.c: Fixed state.changed Fixed that REDO keeps pages pinned until UNDO Some bug fixes from previous commit Fixes for UNDO/REDO of key pages storage/maria/ma_search.c: Fixed packing and storing of keys to provide more information to caller so that we can do efficent REDO logging of the changes. storage/maria/ma_test1.c: Fixed bug with not initialized variable storage/maria/ma_test2.c: Removed not used code storage/maria/ma_test_all.res: Updated results storage/maria/ma_test_all.sh: Changed one test to test more Removed timing tests as not relevant here storage/maria/ma_test_recovery.expected: Updated test result after redo/undo if key pages works storage/maria/ma_test_recovery: Updated test after redo/undo if key pages works storage/maria/ma_write.c: Moved some general log functions to ma_key_recover.c Fixed some bugs in undo Moved ma_log_split() to _ma_split_page() Small changes in some function arguments to be able to do redo logging storage/maria/maria_chk.c: disable logging while doing repair table storage/maria/maria_def.h: New function prototypes Move some structs and functions to ma_key_recover.c storage/maria/unittest/ma_control_file-t.c: Updated with patch from Sanja NOTE: This is not complete and need to be updated to new control file format storage/maria/unittest/ma_test_loghandler-t.c: Fixed compiler warning
2007-11-20 16:42:16 +01:00
/* Struct for clr_end */
struct st_msg_to_write_hook_for_clr_end
{
LSN previous_undo_lsn;
enum translog_record_type undone_record_type;
ha_checksum checksum_delta;
void *extra_msg;
};
struct st_msg_to_write_hook_for_undo_key
{
my_off_t *root;
my_off_t value;
Fixed repair_by_sort to work with BLOCK_RECORD Fixed bugs in undo logging Fixed bug where head block was split before min_row_length (caused Maria to believe row was crashed on read) Reserved place for reference-transid on key pages (for packing of transids) ALTER TABLE and INSERT ... SELECT now uses fast creation of index Known bugs: ma_test_recovery fails because of a bug in redo handling when log is cut directly after a redo (Guilhem knows how to fix) ma_test_recovery.excepted is not totally correct, because of the above bug mysqld sometimes fails to restart; Fails with error "end_of_redo_phase: Assertion `long_trid != 0' failed"; Guilhem to investigate include/maria.h: Prototype changes Added current_filepos to st_maria_sort_info mysql-test/r/maria.result: Updated results that changes as alter table and insert ... select now uses fast creation of index mysys/mf_iocache.c: Reset variable to gurard against double invocation storage/maria/ma_bitmap.c: Added _ma_bitmap_reset_cache() (needed for repair) storage/maria/ma_blockrec.c: Simplify code More initial allocations Fixed bug where head block was split before min_row_length (caused Maria to believe row was crashed on read) storage/maria/ma_blockrec.h: Moved TRANSID_SIZE to maria_def.h Added prototype for new functions storage/maria/ma_check.c: Simplicy code Fixed repair_by_sort to work with BLOCK_RECORD - When using BLOCK_RECORD or UNPACK create new Maria handle - Use common initializer function - Align code with maria_repair() Made some changes to maria_repair_parallel() to use common initializer function Removed ASK_MONTY section by fixing noted problem storage/maria/ma_close.c: Moved check for readonly to _ma_state_info_write() storage/maria/ma_key_recover.c: Use different log entries if key root changes or not. This fixed some bugs when tree grows storage/maria/ma_key_recover.h: Added keynr to st_msg_to_write_hook_for_undo_key storage/maria/ma_loghandler.c: Added INIT_LOGREC_UNDO_KEY_INSERT_WITH_ROOT storage/maria/ma_loghandler.h: Added INIT_LOGREC_UNDO_KEY_INSERT_WITH_ROOT storage/maria/ma_open.c: Added TRANSID to all key pages (for future compressing of trans id's) For compressed records, alloc a bit bigger buffer to avoid valgrind warnings If table is opened readonly, don't update state storage/maria/ma_packrec.c: Allocate bigger array for bit unpacking to avoid valgrind errors storage/maria/ma_recovery.c: Added UNDO_KEY_INSERT_WITH_ROOT & UNDO_KEY_DELETE_WITH_ROOT storage/maria/ma_sort.c: More logging storage/maria/ma_test_all.sh: More tests storage/maria/ma_test_recovery.expected: Update results Note that this is not complete becasue of a bug in recovery storage/maria/ma_test_recovery: Removed recreation of index (not needed when we have redo for index pages) storage/maria/maria_chk.c: When using flag --read-only, don't update status for files When using --unpack, don't use REPAIR_BY_SORT if other repair option is given Enable repair_by_sort for BLOCK records Removed not needed newline at start of --describe storage/maria/maria_def.h: Support for TRANSID_SIZE to key pages storage/maria/maria_read_log.c: renamed --only-display to --display-only
2007-11-28 20:38:30 +01:00
uint keynr;
WL#3072 - Maria Recovery: recovery of state.auto_increment. When we log UNDO_KEY_INSERT for an auto_inc key, we update state.auto_increment (not anymore at the end of maria_write() except if this is a non-transactional table). When Recovery sees UNDO_KEY_INSERT in the REDO phase, it reads the auto_inc value from it and updates state.auto_increment. mysql-test/r/maria-recovery.result: Without the code fix, there would be in CHECK TABLE: "Auto-increment value: 0 is smaller than max used value: 3" and no AUTO_INCREMENT= clause in SHOW CREATE TABLE. mysql-test/t/maria-recovery.test: Test of recovery of state.auto_increment: from an old table, does the replaying of the log set state.auto_increment to what it should be. storage/maria/ma_check.c: new way of calling ma_retrieve_auto_increment(): pass key storage/maria/ma_key.c: ma_retrieve_auto_increment() now operates directly with a pointer to the key and not on the record. storage/maria/ma_key_recover.c: dedicated write_hook_for_undo_key_insert(): sets state.auto_increment under log's mutex. storage/maria/ma_key_recover.h: Dedicated hook for UNDO_KEY_INSERT, to set state.auto_increment. Such hook needs a new member st_msg_write_hook_for_undo_key::auto_increment, which contains the auto_increment value inserted. storage/maria/ma_loghandler.c: UNDO_KEY_INSERT gets a dedicated write_hook, to set auto_increment. storage/maria/ma_recovery.c: When in the REDO phase we see UNDO_KEY_INSERT: if the state is older than this record, and the key is the auto_increment one, read the key's value from the log record and update state.auto_increment. storage/maria/ma_test_all.sh: use $maria_path to be able to run from /dev/shm (faster) storage/maria/ma_update.c: bool is more of C++, using my_bool. If table is transactional, state.auto_increment is already updated in write_hook_for_undo_key_insert(). storage/maria/ma_write.c: If table is transactional, state.auto_increment is not updated at the end of maria_write() but rather in write_hook_for_undo_key_insert() (under log's mutex, so that a concurrent checkpoint does not read state.auto_increment while it is changing - corrupted). _ma_ck_write_btree_with_log() extracts the auto_increment value from the key, puts it into msg.auto_increment, and this is passed to write_hook_for_undo_key_insert(). storage/maria/maria_def.h: change of prototype of ma_retrieve_auto_increment() storage/maria/maria_read_log.c: use default log file size. Use separate page caches for table and logs (needed if maria_block_size!=TRANSLOG_PAGE_SIZE).
2007-12-12 22:33:36 +01:00
ulonglong auto_increment;
Fixes for redo/undo logging of key pages New extendable format for maria_log_control file Fixed some compiler warnings include/maria.h: Added maria_disable_logging() and maria_enable_logging() mysql-test/include/maria_verify_recovery.inc: Updated tests now when key redo/undo works mysql-test/r/maria-recovery.result: Updated tests now when key redo/undo works storage/maria/ma_blockrec.c: Use unified CLR code Added rec_lsn for full pages Moved clr write hook to ma_key_recover.c Changed REDO code to keep pages pinned until undo Mark page_link's as changed storage/maria/ma_blockrec.h: Moved write_hook_for_clr_end() to ma_key_recover.c storage/maria/ma_check.c: Changed key check code to use PAGECACHE_READ_UNKNOWN_PAGE Fixed wrong warning when checking files after maria_pack When unpacking files, we have to use new keypos_to_recpos method When doing repair, we can disregard index key file pages in page cache storage/maria/ma_commit.c: Added simple enable/disable logging functions (Needed for recovery) storage/maria/ma_control_file.c: Make maria control file extendable without having to make it incompatible for older versions storage/maria/ma_control_file.h: New error messages Added CONTROL_FILE_VERSION storage/maria/ma_delete.c: Added redo/undo for key pages change_length -> changed_length to make things similar More comments & more DBUG storage/maria/ma_key_recover.c: Unified CLR method Moved here write_hook_for_clr_end() and common keypage log functions Changed REDO to keep pages pinned until undo Changed UNDO code to change key_root under log mutex storage/maria/ma_key_recover.h: New structures and functions storage/maria/ma_loghandler.c: Include needed files storage/maria/ma_open.c: Change maria_open() to use pread() instead of read() storage/maria/ma_page.c: Fixed bug in key_del handling Clear pages if IDENTICAL_PAGES_AFTER_RECOVERY is defined storage/maria/ma_pagecache.c: Indentation and spelling fixes More DBUG Added helper function: pagecache_block_link_to_buffer() storage/maria/ma_pagecache.h: Added pagecache_block_link_to_buffer() storage/maria/ma_recovery.c: Fixed state.changed Fixed that REDO keeps pages pinned until UNDO Some bug fixes from previous commit Fixes for UNDO/REDO of key pages storage/maria/ma_search.c: Fixed packing and storing of keys to provide more information to caller so that we can do efficent REDO logging of the changes. storage/maria/ma_test1.c: Fixed bug with not initialized variable storage/maria/ma_test2.c: Removed not used code storage/maria/ma_test_all.res: Updated results storage/maria/ma_test_all.sh: Changed one test to test more Removed timing tests as not relevant here storage/maria/ma_test_recovery.expected: Updated test result after redo/undo if key pages works storage/maria/ma_test_recovery: Updated test after redo/undo if key pages works storage/maria/ma_write.c: Moved some general log functions to ma_key_recover.c Fixed some bugs in undo Moved ma_log_split() to _ma_split_page() Small changes in some function arguments to be able to do redo logging storage/maria/maria_chk.c: disable logging while doing repair table storage/maria/maria_def.h: New function prototypes Move some structs and functions to ma_key_recover.c storage/maria/unittest/ma_control_file-t.c: Updated with patch from Sanja NOTE: This is not complete and need to be updated to new control file format storage/maria/unittest/ma_test_loghandler-t.c: Fixed compiler warning
2007-11-20 16:42:16 +01:00
};
First part of redo/undo for key pages Added key_nr to st_maria_keydef for faster keyinfo->keynr conversion For transactional tables, shift record number in keys up with 1 bit to have place to indicate if transid follows Checksum for MyISAM now ignores NULL and not used part of VARCHAR Renamed some variables that caused shadow compiler warnings Moved extra() call when waiting for tables to not be used to after tables are removed from cache. Fixed crashing bugs when using Maria TEMPORARY tables with TRUNCATE. Removed 'hack' code in sql directory to go around this bug. pagecache_unlock_by_ulink() now has extra argument to say if page was changed. Give error message if we fail to open control file Mark page cache variables as not flushable include/maria.h: Made min page cache larger (needed for pinning key page) Added key_nr to st_maria_keydef for faster keyinfo->keynr conversion Added write_comp_flag to move some runtime code to maria_open() include/my_base.h: Added new error message to be used when handler initialization failed include/my_global.h: Renamed dummy to swap_dummy to avoid conflicts with local 'dummy' variables include/my_handler.h: Added const to some parameters mysys/array.c: More DBUG mysys/my_error.c: Fixed indentation mysys/my_handler.c: Added const to some parameters Added missing error messages sql/field.h: Renamed variables to avoid variable shadowing sql/handler.h: Renamed parameter to avoid variable name conflict sql/item.h: Renamed variables to avoid variable shadowing sql/log_event_old.h: Renamed variables to avoid variable shadowing sql/set_var.h: Renamed variables to avoid variable shadowing sql/sql_delete.cc: Removed maria hack for temporary tables Fixed indentation sql/sql_table.cc: Moved extra() call when waiting for tables to not be used to after tables are removed from cache. This was needed to ensure we don't do a PREPARE_FOR_DROP or similar call while the table is still in use. sql/table.cc: Copy page_checksum from share Removed Maria hack storage/maria/Makefile.am: Added new files storage/maria/ha_maria.cc: Renamed records -> record_count and info -> create_info to avoid variable name conflicts Mark page cache variables as not flushable storage/maria/ma_blockrec.c: Moved _ma_unpin_all_pages() to ma_key_recover.c Moved init of info->pinned_pages to ma_open.c Moved _ma_finalize_row() to maria_key_recover.h Renamed some variables to avoid variable name conflicts Mark page_link.changed for blocks we change directly Simplify handling of undo link when writing LOGREC_UNDO_ROW_INSERT (old code crashed when having redo for index) storage/maria/ma_blockrec.h: Removed extra empty line storage/maria/ma_checkpoint.c: Remove not needed trnman.h storage/maria/ma_close.c: Free pinned pages (which are now always allocated) storage/maria/ma_control_file.c: Give error message if we fail to open control file storage/maria/ma_delete.c: Changes for redo logging (first part, logging of underflow not yet done) - Log undo-key-delete - Log delete of key - Updated arguments to _ma_fetch_keypage(), _ma_dispose(), _ma_write_keypage(), _ma_insert() - Added new arguments to some functions to be able to write redo information - Mark key pages as changed when we write with PAGECACHE_LOCK_LEFT_WRITELOCKED Remove one not needed _ma_write_keypage() in d_search() when upper level will do the write anyway Changed 2 bmove_upp() to bmove() as this made code easer to understand More function comments Indentation fixes storage/maria/ma_ft_update.c: New arguments to _ma_write_keypage() storage/maria/ma_loghandler.c: Fixed some DBUG_PRINT messages Simplify code Added new log entrys for key page redo Renamed some variables to avoid variable name shadowing storage/maria/ma_loghandler.h: Moved some defines here Added define for storing key number on key pages Added new translog record types Added enum for type of operations in LOGREC_REDO_INDEX storage/maria/ma_open.c: Always allocate info.pinned_pages (we need now also for normal key page usage) Update keyinfo->key_nr Added virtual functions to convert record position o number to be stored on key pages Update keyinfo->write_comp_flag to value of search flag to be used when writing key storage/maria/ma_page.c: Added redo for key pages - Extended _ma_fetch_keypage() with type of lock to put on page and address to used MARIA_PINNED_PAGE - _ma_fetch_keypage() now pin's pages if needed - Extended _ma_write_keypage() with type of locks to be used - ma_dispose() now locks info->s->state.key_del from other threads - ma_dispose() writes redo log record - ma_new() locks info->s->state.key_del from other threads if it was used - ma_new() now pins read page Other things: - Removed some not needed arguments from _ma_new() and _ma_dispose) - Added some new variables to simplify code - If EXTRA_DEBUG is used, do crc on full page to catch not unitialized bytes storage/maria/ma_pagecache.h: Applied patch from Sanja to add extra argument to pagecache_unlock_by_ulink() to mark if page was changed Added some defines for pagecache priority levels that one can use storage/maria/ma_range.c: Added new arguments for call to _ma_fetch_keypage() storage/maria/ma_recovery.c: - Added hooks for new translog types: REDO_INDEX, REDO_INDEX_NEW_PAGE, REDO_INDEX_FREE_PAGE, UNDO_KEY_INSERT, UNDO_KEY_DELETE and UNDO_KEY_DELETE_WITH_ROOT. - Moved variable declarations to start of function (portability fixes) - Removed some not needed initializations - Set only relevant state changes for each redo/undo entry storage/maria/lockman.c: Removed end space storage/maria/ma_check.c: Removed end space storage/maria/ma_create.c: Removed end space storage/maria/ma_locking.c: Removed end space storage/maria/ma_packrec.c: Removed end space storage/maria/ma_pagecache.c: Removed end space storage/maria/ma_panic.c: Removed end space storage/maria/ma_rt_index.c: Added new arguments for call to _ma_fetch_keypage(), _ma_write_keypage(), _ma_dispose() and _ma_new() Fixed indentation storage/maria/ma_rt_key.c: Added new arguments for call to _ma_fetch_keypage() storage/maria/ma_rt_split.c: Added new arguments for call to _ma_new() Use new keypage header Added new arguments for call to _ma_write_keypage() storage/maria/ma_search.c: Updated comments & indentation Added new arguments for call to _ma_fetch_keypage() Made some variables and arguments const Added virtual functions for converting row position to number to be stored in key use MARIA_RECORD_POS of record position instead of my_off_t Record in MARIA_KEY_PARAM how page was changed one key insert (needed for REDO) storage/maria/ma_sort.c: Removed end space storage/maria/ma_statrec.c: Updated arguments for call to _ma_rec_pos() storage/maria/ma_test1.c: Fixed too small buffer to init_pagecache() Fixed bug when using insert_count and test_flag storage/maria/ma_test2.c: Use more resonable pagecache size Remove not used code Reset blob_length to fix wrong output message storage/maria/ma_test_all.sh: Fixed wrong test storage/maria/ma_write.c: Lots of new code to handle REDO of key pages No logic changes because of REDO code, mostly adding new arguments and adding new code for logging Added new arguments for calls to _ma_fetch_keypage(), _ma_write_keypage() and similar functions Move setting of comp_flag in ma_ck_wrte_btree() from runtime to maria_open() Zerofill new used pages for: - To remove possible sensitive data left in buffer - To get idenitical data on pages after running redo - Better compression of pages if archived storage/maria/maria_chk.c: Added information if table is crash safe storage/maria/maria_def.h: New virtual function to convert between record position on key and normal record position Aded mutex and extra variables to handle locking of share->state.key_del Moved some structure variables to get things more aligned Added extra arguments to MARIA_KEY_PARAM to be able to remember what was changed on key page on key insert Added argument to MARIA_PINNED_PAGE to indicate if page was changed Updated prototypes for functions Added some structures for signaling changes in REDO handling storage/maria/unittest/ma_pagecache_single.c: Updated arguments for changed function calls storage/myisam/mi_check.c: Made calc_check_checksum virtual storage/myisam/mi_checksum.c: Update checksums to ignore null columns storage/myisam/mi_create.c: Mark if table has null column (to know when we have to use mi_checksum()) storage/myisam/mi_open.c: Added virtual function for calculating checksum to be able to easily ignore NULL fields storage/myisam/mi_test2.c: Fixed bug storage/myisam/myisamdef.h: Added virtual function for calculating checksum during check table Removed ha_key_cmp() as this is in handler.h storage/maria/ma_key_recover.c: New BitKeeper file ``storage/maria/ma_key_recover.c'' storage/maria/ma_key_recover.h: New BitKeeper file ``storage/maria/ma_key_recover.h'' storage/maria/ma_key_redo.c: New BitKeeper file ``storage/maria/ma_key_redo.c''
2007-11-14 18:08:06 +01:00
/* Function definitions for some redo functions */
my_bool _ma_write_clr(MARIA_HA *info, LSN undo_lsn,
enum translog_record_type undo_type,
my_bool store_checksum, ha_checksum checksum,
Fixes for redo/undo logging of key pages New extendable format for maria_log_control file Fixed some compiler warnings include/maria.h: Added maria_disable_logging() and maria_enable_logging() mysql-test/include/maria_verify_recovery.inc: Updated tests now when key redo/undo works mysql-test/r/maria-recovery.result: Updated tests now when key redo/undo works storage/maria/ma_blockrec.c: Use unified CLR code Added rec_lsn for full pages Moved clr write hook to ma_key_recover.c Changed REDO code to keep pages pinned until undo Mark page_link's as changed storage/maria/ma_blockrec.h: Moved write_hook_for_clr_end() to ma_key_recover.c storage/maria/ma_check.c: Changed key check code to use PAGECACHE_READ_UNKNOWN_PAGE Fixed wrong warning when checking files after maria_pack When unpacking files, we have to use new keypos_to_recpos method When doing repair, we can disregard index key file pages in page cache storage/maria/ma_commit.c: Added simple enable/disable logging functions (Needed for recovery) storage/maria/ma_control_file.c: Make maria control file extendable without having to make it incompatible for older versions storage/maria/ma_control_file.h: New error messages Added CONTROL_FILE_VERSION storage/maria/ma_delete.c: Added redo/undo for key pages change_length -> changed_length to make things similar More comments & more DBUG storage/maria/ma_key_recover.c: Unified CLR method Moved here write_hook_for_clr_end() and common keypage log functions Changed REDO to keep pages pinned until undo Changed UNDO code to change key_root under log mutex storage/maria/ma_key_recover.h: New structures and functions storage/maria/ma_loghandler.c: Include needed files storage/maria/ma_open.c: Change maria_open() to use pread() instead of read() storage/maria/ma_page.c: Fixed bug in key_del handling Clear pages if IDENTICAL_PAGES_AFTER_RECOVERY is defined storage/maria/ma_pagecache.c: Indentation and spelling fixes More DBUG Added helper function: pagecache_block_link_to_buffer() storage/maria/ma_pagecache.h: Added pagecache_block_link_to_buffer() storage/maria/ma_recovery.c: Fixed state.changed Fixed that REDO keeps pages pinned until UNDO Some bug fixes from previous commit Fixes for UNDO/REDO of key pages storage/maria/ma_search.c: Fixed packing and storing of keys to provide more information to caller so that we can do efficent REDO logging of the changes. storage/maria/ma_test1.c: Fixed bug with not initialized variable storage/maria/ma_test2.c: Removed not used code storage/maria/ma_test_all.res: Updated results storage/maria/ma_test_all.sh: Changed one test to test more Removed timing tests as not relevant here storage/maria/ma_test_recovery.expected: Updated test result after redo/undo if key pages works storage/maria/ma_test_recovery: Updated test after redo/undo if key pages works storage/maria/ma_write.c: Moved some general log functions to ma_key_recover.c Fixed some bugs in undo Moved ma_log_split() to _ma_split_page() Small changes in some function arguments to be able to do redo logging storage/maria/maria_chk.c: disable logging while doing repair table storage/maria/maria_def.h: New function prototypes Move some structs and functions to ma_key_recover.c storage/maria/unittest/ma_control_file-t.c: Updated with patch from Sanja NOTE: This is not complete and need to be updated to new control file format storage/maria/unittest/ma_test_loghandler-t.c: Fixed compiler warning
2007-11-20 16:42:16 +01:00
LSN *res_lsn, void *extra_msg);
my_bool write_hook_for_clr_end(enum translog_record_type type,
TRN *trn, MARIA_HA *tbl_info, LSN *lsn,
void *hook_arg);
extern my_bool write_hook_for_undo_key(enum translog_record_type type,
TRN *trn, MARIA_HA *tbl_info,
LSN *lsn, void *hook_arg);
WL#3072 - Maria Recovery: recovery of state.auto_increment. When we log UNDO_KEY_INSERT for an auto_inc key, we update state.auto_increment (not anymore at the end of maria_write() except if this is a non-transactional table). When Recovery sees UNDO_KEY_INSERT in the REDO phase, it reads the auto_inc value from it and updates state.auto_increment. mysql-test/r/maria-recovery.result: Without the code fix, there would be in CHECK TABLE: "Auto-increment value: 0 is smaller than max used value: 3" and no AUTO_INCREMENT= clause in SHOW CREATE TABLE. mysql-test/t/maria-recovery.test: Test of recovery of state.auto_increment: from an old table, does the replaying of the log set state.auto_increment to what it should be. storage/maria/ma_check.c: new way of calling ma_retrieve_auto_increment(): pass key storage/maria/ma_key.c: ma_retrieve_auto_increment() now operates directly with a pointer to the key and not on the record. storage/maria/ma_key_recover.c: dedicated write_hook_for_undo_key_insert(): sets state.auto_increment under log's mutex. storage/maria/ma_key_recover.h: Dedicated hook for UNDO_KEY_INSERT, to set state.auto_increment. Such hook needs a new member st_msg_write_hook_for_undo_key::auto_increment, which contains the auto_increment value inserted. storage/maria/ma_loghandler.c: UNDO_KEY_INSERT gets a dedicated write_hook, to set auto_increment. storage/maria/ma_recovery.c: When in the REDO phase we see UNDO_KEY_INSERT: if the state is older than this record, and the key is the auto_increment one, read the key's value from the log record and update state.auto_increment. storage/maria/ma_test_all.sh: use $maria_path to be able to run from /dev/shm (faster) storage/maria/ma_update.c: bool is more of C++, using my_bool. If table is transactional, state.auto_increment is already updated in write_hook_for_undo_key_insert(). storage/maria/ma_write.c: If table is transactional, state.auto_increment is not updated at the end of maria_write() but rather in write_hook_for_undo_key_insert() (under log's mutex, so that a concurrent checkpoint does not read state.auto_increment while it is changing - corrupted). _ma_ck_write_btree_with_log() extracts the auto_increment value from the key, puts it into msg.auto_increment, and this is passed to write_hook_for_undo_key_insert(). storage/maria/maria_def.h: change of prototype of ma_retrieve_auto_increment() storage/maria/maria_read_log.c: use default log file size. Use separate page caches for table and logs (needed if maria_block_size!=TRANSLOG_PAGE_SIZE).
2007-12-12 22:33:36 +01:00
extern my_bool write_hook_for_undo_key_insert(enum translog_record_type type,
TRN *trn, MARIA_HA *tbl_info,
LSN *lsn, void *hook_arg);
First part of redo/undo for key pages Added key_nr to st_maria_keydef for faster keyinfo->keynr conversion For transactional tables, shift record number in keys up with 1 bit to have place to indicate if transid follows Checksum for MyISAM now ignores NULL and not used part of VARCHAR Renamed some variables that caused shadow compiler warnings Moved extra() call when waiting for tables to not be used to after tables are removed from cache. Fixed crashing bugs when using Maria TEMPORARY tables with TRUNCATE. Removed 'hack' code in sql directory to go around this bug. pagecache_unlock_by_ulink() now has extra argument to say if page was changed. Give error message if we fail to open control file Mark page cache variables as not flushable include/maria.h: Made min page cache larger (needed for pinning key page) Added key_nr to st_maria_keydef for faster keyinfo->keynr conversion Added write_comp_flag to move some runtime code to maria_open() include/my_base.h: Added new error message to be used when handler initialization failed include/my_global.h: Renamed dummy to swap_dummy to avoid conflicts with local 'dummy' variables include/my_handler.h: Added const to some parameters mysys/array.c: More DBUG mysys/my_error.c: Fixed indentation mysys/my_handler.c: Added const to some parameters Added missing error messages sql/field.h: Renamed variables to avoid variable shadowing sql/handler.h: Renamed parameter to avoid variable name conflict sql/item.h: Renamed variables to avoid variable shadowing sql/log_event_old.h: Renamed variables to avoid variable shadowing sql/set_var.h: Renamed variables to avoid variable shadowing sql/sql_delete.cc: Removed maria hack for temporary tables Fixed indentation sql/sql_table.cc: Moved extra() call when waiting for tables to not be used to after tables are removed from cache. This was needed to ensure we don't do a PREPARE_FOR_DROP or similar call while the table is still in use. sql/table.cc: Copy page_checksum from share Removed Maria hack storage/maria/Makefile.am: Added new files storage/maria/ha_maria.cc: Renamed records -> record_count and info -> create_info to avoid variable name conflicts Mark page cache variables as not flushable storage/maria/ma_blockrec.c: Moved _ma_unpin_all_pages() to ma_key_recover.c Moved init of info->pinned_pages to ma_open.c Moved _ma_finalize_row() to maria_key_recover.h Renamed some variables to avoid variable name conflicts Mark page_link.changed for blocks we change directly Simplify handling of undo link when writing LOGREC_UNDO_ROW_INSERT (old code crashed when having redo for index) storage/maria/ma_blockrec.h: Removed extra empty line storage/maria/ma_checkpoint.c: Remove not needed trnman.h storage/maria/ma_close.c: Free pinned pages (which are now always allocated) storage/maria/ma_control_file.c: Give error message if we fail to open control file storage/maria/ma_delete.c: Changes for redo logging (first part, logging of underflow not yet done) - Log undo-key-delete - Log delete of key - Updated arguments to _ma_fetch_keypage(), _ma_dispose(), _ma_write_keypage(), _ma_insert() - Added new arguments to some functions to be able to write redo information - Mark key pages as changed when we write with PAGECACHE_LOCK_LEFT_WRITELOCKED Remove one not needed _ma_write_keypage() in d_search() when upper level will do the write anyway Changed 2 bmove_upp() to bmove() as this made code easer to understand More function comments Indentation fixes storage/maria/ma_ft_update.c: New arguments to _ma_write_keypage() storage/maria/ma_loghandler.c: Fixed some DBUG_PRINT messages Simplify code Added new log entrys for key page redo Renamed some variables to avoid variable name shadowing storage/maria/ma_loghandler.h: Moved some defines here Added define for storing key number on key pages Added new translog record types Added enum for type of operations in LOGREC_REDO_INDEX storage/maria/ma_open.c: Always allocate info.pinned_pages (we need now also for normal key page usage) Update keyinfo->key_nr Added virtual functions to convert record position o number to be stored on key pages Update keyinfo->write_comp_flag to value of search flag to be used when writing key storage/maria/ma_page.c: Added redo for key pages - Extended _ma_fetch_keypage() with type of lock to put on page and address to used MARIA_PINNED_PAGE - _ma_fetch_keypage() now pin's pages if needed - Extended _ma_write_keypage() with type of locks to be used - ma_dispose() now locks info->s->state.key_del from other threads - ma_dispose() writes redo log record - ma_new() locks info->s->state.key_del from other threads if it was used - ma_new() now pins read page Other things: - Removed some not needed arguments from _ma_new() and _ma_dispose) - Added some new variables to simplify code - If EXTRA_DEBUG is used, do crc on full page to catch not unitialized bytes storage/maria/ma_pagecache.h: Applied patch from Sanja to add extra argument to pagecache_unlock_by_ulink() to mark if page was changed Added some defines for pagecache priority levels that one can use storage/maria/ma_range.c: Added new arguments for call to _ma_fetch_keypage() storage/maria/ma_recovery.c: - Added hooks for new translog types: REDO_INDEX, REDO_INDEX_NEW_PAGE, REDO_INDEX_FREE_PAGE, UNDO_KEY_INSERT, UNDO_KEY_DELETE and UNDO_KEY_DELETE_WITH_ROOT. - Moved variable declarations to start of function (portability fixes) - Removed some not needed initializations - Set only relevant state changes for each redo/undo entry storage/maria/lockman.c: Removed end space storage/maria/ma_check.c: Removed end space storage/maria/ma_create.c: Removed end space storage/maria/ma_locking.c: Removed end space storage/maria/ma_packrec.c: Removed end space storage/maria/ma_pagecache.c: Removed end space storage/maria/ma_panic.c: Removed end space storage/maria/ma_rt_index.c: Added new arguments for call to _ma_fetch_keypage(), _ma_write_keypage(), _ma_dispose() and _ma_new() Fixed indentation storage/maria/ma_rt_key.c: Added new arguments for call to _ma_fetch_keypage() storage/maria/ma_rt_split.c: Added new arguments for call to _ma_new() Use new keypage header Added new arguments for call to _ma_write_keypage() storage/maria/ma_search.c: Updated comments & indentation Added new arguments for call to _ma_fetch_keypage() Made some variables and arguments const Added virtual functions for converting row position to number to be stored in key use MARIA_RECORD_POS of record position instead of my_off_t Record in MARIA_KEY_PARAM how page was changed one key insert (needed for REDO) storage/maria/ma_sort.c: Removed end space storage/maria/ma_statrec.c: Updated arguments for call to _ma_rec_pos() storage/maria/ma_test1.c: Fixed too small buffer to init_pagecache() Fixed bug when using insert_count and test_flag storage/maria/ma_test2.c: Use more resonable pagecache size Remove not used code Reset blob_length to fix wrong output message storage/maria/ma_test_all.sh: Fixed wrong test storage/maria/ma_write.c: Lots of new code to handle REDO of key pages No logic changes because of REDO code, mostly adding new arguments and adding new code for logging Added new arguments for calls to _ma_fetch_keypage(), _ma_write_keypage() and similar functions Move setting of comp_flag in ma_ck_wrte_btree() from runtime to maria_open() Zerofill new used pages for: - To remove possible sensitive data left in buffer - To get idenitical data on pages after running redo - Better compression of pages if archived storage/maria/maria_chk.c: Added information if table is crash safe storage/maria/maria_def.h: New virtual function to convert between record position on key and normal record position Aded mutex and extra variables to handle locking of share->state.key_del Moved some structure variables to get things more aligned Added extra arguments to MARIA_KEY_PARAM to be able to remember what was changed on key page on key insert Added argument to MARIA_PINNED_PAGE to indicate if page was changed Updated prototypes for functions Added some structures for signaling changes in REDO handling storage/maria/unittest/ma_pagecache_single.c: Updated arguments for changed function calls storage/myisam/mi_check.c: Made calc_check_checksum virtual storage/myisam/mi_checksum.c: Update checksums to ignore null columns storage/myisam/mi_create.c: Mark if table has null column (to know when we have to use mi_checksum()) storage/myisam/mi_open.c: Added virtual function for calculating checksum to be able to easily ignore NULL fields storage/myisam/mi_test2.c: Fixed bug storage/myisam/myisamdef.h: Added virtual function for calculating checksum during check table Removed ha_key_cmp() as this is in handler.h storage/maria/ma_key_recover.c: New BitKeeper file ``storage/maria/ma_key_recover.c'' storage/maria/ma_key_recover.h: New BitKeeper file ``storage/maria/ma_key_recover.h'' storage/maria/ma_key_redo.c: New BitKeeper file ``storage/maria/ma_key_redo.c''
2007-11-14 18:08:06 +01:00
void _ma_unpin_all_pages(MARIA_HA *info, LSN undo_lsn);
Fixes for redo/undo logging of key pages New extendable format for maria_log_control file Fixed some compiler warnings include/maria.h: Added maria_disable_logging() and maria_enable_logging() mysql-test/include/maria_verify_recovery.inc: Updated tests now when key redo/undo works mysql-test/r/maria-recovery.result: Updated tests now when key redo/undo works storage/maria/ma_blockrec.c: Use unified CLR code Added rec_lsn for full pages Moved clr write hook to ma_key_recover.c Changed REDO code to keep pages pinned until undo Mark page_link's as changed storage/maria/ma_blockrec.h: Moved write_hook_for_clr_end() to ma_key_recover.c storage/maria/ma_check.c: Changed key check code to use PAGECACHE_READ_UNKNOWN_PAGE Fixed wrong warning when checking files after maria_pack When unpacking files, we have to use new keypos_to_recpos method When doing repair, we can disregard index key file pages in page cache storage/maria/ma_commit.c: Added simple enable/disable logging functions (Needed for recovery) storage/maria/ma_control_file.c: Make maria control file extendable without having to make it incompatible for older versions storage/maria/ma_control_file.h: New error messages Added CONTROL_FILE_VERSION storage/maria/ma_delete.c: Added redo/undo for key pages change_length -> changed_length to make things similar More comments & more DBUG storage/maria/ma_key_recover.c: Unified CLR method Moved here write_hook_for_clr_end() and common keypage log functions Changed REDO to keep pages pinned until undo Changed UNDO code to change key_root under log mutex storage/maria/ma_key_recover.h: New structures and functions storage/maria/ma_loghandler.c: Include needed files storage/maria/ma_open.c: Change maria_open() to use pread() instead of read() storage/maria/ma_page.c: Fixed bug in key_del handling Clear pages if IDENTICAL_PAGES_AFTER_RECOVERY is defined storage/maria/ma_pagecache.c: Indentation and spelling fixes More DBUG Added helper function: pagecache_block_link_to_buffer() storage/maria/ma_pagecache.h: Added pagecache_block_link_to_buffer() storage/maria/ma_recovery.c: Fixed state.changed Fixed that REDO keeps pages pinned until UNDO Some bug fixes from previous commit Fixes for UNDO/REDO of key pages storage/maria/ma_search.c: Fixed packing and storing of keys to provide more information to caller so that we can do efficent REDO logging of the changes. storage/maria/ma_test1.c: Fixed bug with not initialized variable storage/maria/ma_test2.c: Removed not used code storage/maria/ma_test_all.res: Updated results storage/maria/ma_test_all.sh: Changed one test to test more Removed timing tests as not relevant here storage/maria/ma_test_recovery.expected: Updated test result after redo/undo if key pages works storage/maria/ma_test_recovery: Updated test after redo/undo if key pages works storage/maria/ma_write.c: Moved some general log functions to ma_key_recover.c Fixed some bugs in undo Moved ma_log_split() to _ma_split_page() Small changes in some function arguments to be able to do redo logging storage/maria/maria_chk.c: disable logging while doing repair table storage/maria/maria_def.h: New function prototypes Move some structs and functions to ma_key_recover.c storage/maria/unittest/ma_control_file-t.c: Updated with patch from Sanja NOTE: This is not complete and need to be updated to new control file format storage/maria/unittest/ma_test_loghandler-t.c: Fixed compiler warning
2007-11-20 16:42:16 +01:00
my_bool _ma_log_prefix(MARIA_HA *info, my_off_t page,
uchar *buff, uint changed_length,
int move_length);
my_bool _ma_log_suffix(MARIA_HA *info, my_off_t page,
uchar *buff, uint org_length,
uint new_length);
my_bool _ma_log_add(MARIA_HA *info, my_off_t page, uchar *buff,
uint buff_length, uchar *key_pos,
uint changed_length, int move_length,
my_bool handle_overflow);
First part of redo/undo for key pages Added key_nr to st_maria_keydef for faster keyinfo->keynr conversion For transactional tables, shift record number in keys up with 1 bit to have place to indicate if transid follows Checksum for MyISAM now ignores NULL and not used part of VARCHAR Renamed some variables that caused shadow compiler warnings Moved extra() call when waiting for tables to not be used to after tables are removed from cache. Fixed crashing bugs when using Maria TEMPORARY tables with TRUNCATE. Removed 'hack' code in sql directory to go around this bug. pagecache_unlock_by_ulink() now has extra argument to say if page was changed. Give error message if we fail to open control file Mark page cache variables as not flushable include/maria.h: Made min page cache larger (needed for pinning key page) Added key_nr to st_maria_keydef for faster keyinfo->keynr conversion Added write_comp_flag to move some runtime code to maria_open() include/my_base.h: Added new error message to be used when handler initialization failed include/my_global.h: Renamed dummy to swap_dummy to avoid conflicts with local 'dummy' variables include/my_handler.h: Added const to some parameters mysys/array.c: More DBUG mysys/my_error.c: Fixed indentation mysys/my_handler.c: Added const to some parameters Added missing error messages sql/field.h: Renamed variables to avoid variable shadowing sql/handler.h: Renamed parameter to avoid variable name conflict sql/item.h: Renamed variables to avoid variable shadowing sql/log_event_old.h: Renamed variables to avoid variable shadowing sql/set_var.h: Renamed variables to avoid variable shadowing sql/sql_delete.cc: Removed maria hack for temporary tables Fixed indentation sql/sql_table.cc: Moved extra() call when waiting for tables to not be used to after tables are removed from cache. This was needed to ensure we don't do a PREPARE_FOR_DROP or similar call while the table is still in use. sql/table.cc: Copy page_checksum from share Removed Maria hack storage/maria/Makefile.am: Added new files storage/maria/ha_maria.cc: Renamed records -> record_count and info -> create_info to avoid variable name conflicts Mark page cache variables as not flushable storage/maria/ma_blockrec.c: Moved _ma_unpin_all_pages() to ma_key_recover.c Moved init of info->pinned_pages to ma_open.c Moved _ma_finalize_row() to maria_key_recover.h Renamed some variables to avoid variable name conflicts Mark page_link.changed for blocks we change directly Simplify handling of undo link when writing LOGREC_UNDO_ROW_INSERT (old code crashed when having redo for index) storage/maria/ma_blockrec.h: Removed extra empty line storage/maria/ma_checkpoint.c: Remove not needed trnman.h storage/maria/ma_close.c: Free pinned pages (which are now always allocated) storage/maria/ma_control_file.c: Give error message if we fail to open control file storage/maria/ma_delete.c: Changes for redo logging (first part, logging of underflow not yet done) - Log undo-key-delete - Log delete of key - Updated arguments to _ma_fetch_keypage(), _ma_dispose(), _ma_write_keypage(), _ma_insert() - Added new arguments to some functions to be able to write redo information - Mark key pages as changed when we write with PAGECACHE_LOCK_LEFT_WRITELOCKED Remove one not needed _ma_write_keypage() in d_search() when upper level will do the write anyway Changed 2 bmove_upp() to bmove() as this made code easer to understand More function comments Indentation fixes storage/maria/ma_ft_update.c: New arguments to _ma_write_keypage() storage/maria/ma_loghandler.c: Fixed some DBUG_PRINT messages Simplify code Added new log entrys for key page redo Renamed some variables to avoid variable name shadowing storage/maria/ma_loghandler.h: Moved some defines here Added define for storing key number on key pages Added new translog record types Added enum for type of operations in LOGREC_REDO_INDEX storage/maria/ma_open.c: Always allocate info.pinned_pages (we need now also for normal key page usage) Update keyinfo->key_nr Added virtual functions to convert record position o number to be stored on key pages Update keyinfo->write_comp_flag to value of search flag to be used when writing key storage/maria/ma_page.c: Added redo for key pages - Extended _ma_fetch_keypage() with type of lock to put on page and address to used MARIA_PINNED_PAGE - _ma_fetch_keypage() now pin's pages if needed - Extended _ma_write_keypage() with type of locks to be used - ma_dispose() now locks info->s->state.key_del from other threads - ma_dispose() writes redo log record - ma_new() locks info->s->state.key_del from other threads if it was used - ma_new() now pins read page Other things: - Removed some not needed arguments from _ma_new() and _ma_dispose) - Added some new variables to simplify code - If EXTRA_DEBUG is used, do crc on full page to catch not unitialized bytes storage/maria/ma_pagecache.h: Applied patch from Sanja to add extra argument to pagecache_unlock_by_ulink() to mark if page was changed Added some defines for pagecache priority levels that one can use storage/maria/ma_range.c: Added new arguments for call to _ma_fetch_keypage() storage/maria/ma_recovery.c: - Added hooks for new translog types: REDO_INDEX, REDO_INDEX_NEW_PAGE, REDO_INDEX_FREE_PAGE, UNDO_KEY_INSERT, UNDO_KEY_DELETE and UNDO_KEY_DELETE_WITH_ROOT. - Moved variable declarations to start of function (portability fixes) - Removed some not needed initializations - Set only relevant state changes for each redo/undo entry storage/maria/lockman.c: Removed end space storage/maria/ma_check.c: Removed end space storage/maria/ma_create.c: Removed end space storage/maria/ma_locking.c: Removed end space storage/maria/ma_packrec.c: Removed end space storage/maria/ma_pagecache.c: Removed end space storage/maria/ma_panic.c: Removed end space storage/maria/ma_rt_index.c: Added new arguments for call to _ma_fetch_keypage(), _ma_write_keypage(), _ma_dispose() and _ma_new() Fixed indentation storage/maria/ma_rt_key.c: Added new arguments for call to _ma_fetch_keypage() storage/maria/ma_rt_split.c: Added new arguments for call to _ma_new() Use new keypage header Added new arguments for call to _ma_write_keypage() storage/maria/ma_search.c: Updated comments & indentation Added new arguments for call to _ma_fetch_keypage() Made some variables and arguments const Added virtual functions for converting row position to number to be stored in key use MARIA_RECORD_POS of record position instead of my_off_t Record in MARIA_KEY_PARAM how page was changed one key insert (needed for REDO) storage/maria/ma_sort.c: Removed end space storage/maria/ma_statrec.c: Updated arguments for call to _ma_rec_pos() storage/maria/ma_test1.c: Fixed too small buffer to init_pagecache() Fixed bug when using insert_count and test_flag storage/maria/ma_test2.c: Use more resonable pagecache size Remove not used code Reset blob_length to fix wrong output message storage/maria/ma_test_all.sh: Fixed wrong test storage/maria/ma_write.c: Lots of new code to handle REDO of key pages No logic changes because of REDO code, mostly adding new arguments and adding new code for logging Added new arguments for calls to _ma_fetch_keypage(), _ma_write_keypage() and similar functions Move setting of comp_flag in ma_ck_wrte_btree() from runtime to maria_open() Zerofill new used pages for: - To remove possible sensitive data left in buffer - To get idenitical data on pages after running redo - Better compression of pages if archived storage/maria/maria_chk.c: Added information if table is crash safe storage/maria/maria_def.h: New virtual function to convert between record position on key and normal record position Aded mutex and extra variables to handle locking of share->state.key_del Moved some structure variables to get things more aligned Added extra arguments to MARIA_KEY_PARAM to be able to remember what was changed on key page on key insert Added argument to MARIA_PINNED_PAGE to indicate if page was changed Updated prototypes for functions Added some structures for signaling changes in REDO handling storage/maria/unittest/ma_pagecache_single.c: Updated arguments for changed function calls storage/myisam/mi_check.c: Made calc_check_checksum virtual storage/myisam/mi_checksum.c: Update checksums to ignore null columns storage/myisam/mi_create.c: Mark if table has null column (to know when we have to use mi_checksum()) storage/myisam/mi_open.c: Added virtual function for calculating checksum to be able to easily ignore NULL fields storage/myisam/mi_test2.c: Fixed bug storage/myisam/myisamdef.h: Added virtual function for calculating checksum during check table Removed ha_key_cmp() as this is in handler.h storage/maria/ma_key_recover.c: New BitKeeper file ``storage/maria/ma_key_recover.c'' storage/maria/ma_key_recover.h: New BitKeeper file ``storage/maria/ma_key_recover.h'' storage/maria/ma_key_redo.c: New BitKeeper file ``storage/maria/ma_key_redo.c''
2007-11-14 18:08:06 +01:00
uint _ma_apply_redo_index_new_page(MARIA_HA *info, LSN lsn,
const uchar *header, uint length);
uint _ma_apply_redo_index_free_page(MARIA_HA *info, LSN lsn,
const uchar *header);
uint _ma_apply_redo_index(MARIA_HA *info,
LSN lsn, const uchar *header, uint length);
my_bool _ma_apply_undo_key_insert(MARIA_HA *info, LSN undo_lsn,
const uchar *header, uint length);
my_bool _ma_apply_undo_key_delete(MARIA_HA *info, LSN undo_lsn,
WL#3072 - Maria Recovery Bulk insert: don't log REDO/UNDO for rows, log one UNDO which will truncate files; this is an optimization and a bugfix (table was left half-repaired by crash). Repair: mark table crashed-on-repair at start, bump skip_redo_lsn at start, this is easier for recovery (tells it to skip old REDOs or even UNDO phase) and user (tells it to repair) in case of crash, sync files in the end. Recovery skips missing or corrupted table and moves to next record (in REDO or UNDO phase) to be more robust; warns if happens in UNDO phase. Bugfix for UNDO_KEY_DELETE_WITH_ROOT (tested in ma_test_recovery) and maria_enable_indexes(). Create missing bitmaps when needed (there can be more than one to create, in rare cases), log a record for this. include/myisamchk.h: new flag: bulk insert repair mustn't bump create_rename_lsn mysql-test/lib/mtr_report.pl: skip normal warning in maria-recovery.test mysql-test/r/maria-recovery.result: result: crash before bulk insert is committed, causes proper rollback, and crash right after OPTIMIZE replaces index file with new index file leads to table marked corrupted and recovery not failing. mysql-test/t/maria-recovery.test: - can't check the table or it would commit the transaction, but check is made after recovery. - test of crash before bulk-insert-with-repair is committed (to see if it is rolled back), and of crash after OPTIMIZE has replaced index file but not finished all operations (to see if recovery fails - it used to assert when trying to execute an old REDO on the new index). storage/maria/CMakeLists.txt: new file storage/maria/Makefile.am: new file storage/maria/ha_maria.cc: - If bulk insert on a transactional table using an index repair: table is initially empty, so don't log REDO/UNDO for data rows (optimization), just log an UNDO_BULK_INSERT_WITH_REPAIR which will, if executed, empty the data and index file. Re-enable logging in end_bulk_insert(). - write log record for repair operation only after it's fully done, index sort including (maria_repair*() used to write the log record). - Adding back file->trn=NULL which was removed by mistake earlier. storage/maria/ha_maria.h: new member (see ha_maria.cc) storage/maria/ma_bitmap.c: Functions to create missing bitmaps: - one function which creates missing bitmaps in page cache, except the missing one with max offset which it does not put into page cache as it will be modified very soon. - one function which the one above calls, and creates bitmaps in page cache - one function to execute REDO_BITMAP_NEW_PAGE which uses the second one above. storage/maria/ma_blockrec.c: - when logging REDO_DELETE_ALL, not only 'records' and 'checksum' has to be reset under log's mutex. - execution of REDO_INSERT_ROW_BLOBS now checks the dirty pages' list - execution of UNDO_BULK_INSERT_WITH_REPAIR storage/maria/ma_blockrec.h: new functions storage/maria/ma_check.c: - table-flush-before-repair is moved to a separate function reused by maria_sort_index(); syncing is added - maria_repair() is allowed to re-enable logging only if it is the one which disabled it. - "_ma_flush_table_files_after_repair" was a bad name, it's not after repair now, and it should not sync as we do more changes to the files shortly after (sync is postponed to when writing the log record) - REDO_REPAIR record should be written only after all repair operations (in particular after sorting index in ha_mara::repair()) - close to the end of repair by sort, flushing of pages must happen also in the non-quick case, to prepare for the sync at end. - in parallel repair, some page flushes are not needed as done by initialize_variables_for_repair(). storage/maria/ma_create.c: Update skip_redo_lsn, create_rename_lsn optionally. storage/maria/ma_delete_all.c: Need to sync files at end of maria_delete_all_rows(), if transactional. storage/maria/ma_extra.c: During repair, we sometimes call _ma_flush_table_files() (via _ma_flush_table_files_before_swap()) while there is a WRITE_CACHE. storage/maria/ma_key_recover.c: - when we see CLR_END for UNDO_BULK_INSERT_WITH_REPAIR, re-enable indices. - fixing bug: _ma_apply_undo_key_delete() parsed UNDO_KEY_DELETE_WITH_ROOT wrongly, leading to recovery failure storage/maria/ma_key_recover.h: new prototype storage/maria/ma_locking.c: DBUG_VOID_RETURN missing storage/maria/ma_loghandler.c: UNDO for bulk insert with repair, and REDO for creating bitmaps. LOGREC_FIRST_FREE to not have to change the for() every time we add a new record type. storage/maria/ma_loghandler.h: new UNDO and REDO storage/maria/ma_open.c: Move share.kfile.file=kfile up a bit, so that _ma_update_state_lsns() can get its value, this fixes a bug where LSN_REPAIRED_BY_MARIA_CHK was not corrected on disk by maria_open(). Store skip_redo_lsn in index' header. maria_enable_indexes() had a bug for BLOCK_RECORD, where an empty file has one page, not 0 bytes. storage/maria/ma_recovery.c: - Skip a corrupted, missing, or repaired-with-maria_chk, table in recovery: don't fail, just go to next REDO or UNDO; but if an UNDO is skipped in UNDO phase we issue warnings. - Skip REDO|UNDO in REDO phase if <skip_redo_lsn. - If UNDO phase fails, delete transactions to not make trnman assert. - Update skip_redo_lsn when playing REDO_CREATE_TABLE - Don't record UNDOs for old transactions which we don't know (long_trid==0) - Bugfix for UNDO_KEY_DELETE_WITH_ROOT (see ma_key_recover.c) - Execution of UNDO_BULK_INSERT_WITH_REPAIR - Don't try to find a page number in REDO_DELETE_ALL - Pieces moved to ma_recovery_util.c storage/maria/ma_rename.c: name change storage/maria/ma_static.c: I modified layout of the index' header (inserted skip_redo_lsn in its middle) storage/maria/ma_test2.c: allow breaking the test towards the end, tests execution of UNDO_KEY_DELETE_WITH_ROOT storage/maria/ma_test_recovery.expected: 6 as testflag instead of 4 storage/maria/ma_test_recovery: Increase the amount of rollback work to do when testing recovery with ma_test2; this reproduces the UNDO_KEY_DELETE_WITH_ROOT bug. storage/maria/maria_chk.c: skip_redo_lsn should be updated too, for consistency. Write a REDO_REPAIR after all operations (including sort-records) have been done. No reason to flush blocks after maria_chk_data_link() and maria_sort_records(), there is maria_close() in the end. write_log_record() is a function, to not clutter maria_chk(). storage/maria/maria_def.h: New member skip_redo_lsn in the state, and comments storage/maria/maria_pack.c: skip_redo_lsn should be updated too, for consistency storage/maria/ma_recovery_util.c: _ma_redo_not_needed_for_page(), defined in ma_recovery.c, is needed by ma_blockrec.c; this causes link issues, resolved by putting _ma_redo_not_needed_for_page() into a new file (so that it is not in the same file as repair-related objects of ma_recovery.c). storage/maria/ma_recovery_util.h: new file
2008-01-17 23:59:32 +01:00
const uchar *header, uint length,
my_bool with_root);
First part of redo/undo for key pages Added key_nr to st_maria_keydef for faster keyinfo->keynr conversion For transactional tables, shift record number in keys up with 1 bit to have place to indicate if transid follows Checksum for MyISAM now ignores NULL and not used part of VARCHAR Renamed some variables that caused shadow compiler warnings Moved extra() call when waiting for tables to not be used to after tables are removed from cache. Fixed crashing bugs when using Maria TEMPORARY tables with TRUNCATE. Removed 'hack' code in sql directory to go around this bug. pagecache_unlock_by_ulink() now has extra argument to say if page was changed. Give error message if we fail to open control file Mark page cache variables as not flushable include/maria.h: Made min page cache larger (needed for pinning key page) Added key_nr to st_maria_keydef for faster keyinfo->keynr conversion Added write_comp_flag to move some runtime code to maria_open() include/my_base.h: Added new error message to be used when handler initialization failed include/my_global.h: Renamed dummy to swap_dummy to avoid conflicts with local 'dummy' variables include/my_handler.h: Added const to some parameters mysys/array.c: More DBUG mysys/my_error.c: Fixed indentation mysys/my_handler.c: Added const to some parameters Added missing error messages sql/field.h: Renamed variables to avoid variable shadowing sql/handler.h: Renamed parameter to avoid variable name conflict sql/item.h: Renamed variables to avoid variable shadowing sql/log_event_old.h: Renamed variables to avoid variable shadowing sql/set_var.h: Renamed variables to avoid variable shadowing sql/sql_delete.cc: Removed maria hack for temporary tables Fixed indentation sql/sql_table.cc: Moved extra() call when waiting for tables to not be used to after tables are removed from cache. This was needed to ensure we don't do a PREPARE_FOR_DROP or similar call while the table is still in use. sql/table.cc: Copy page_checksum from share Removed Maria hack storage/maria/Makefile.am: Added new files storage/maria/ha_maria.cc: Renamed records -> record_count and info -> create_info to avoid variable name conflicts Mark page cache variables as not flushable storage/maria/ma_blockrec.c: Moved _ma_unpin_all_pages() to ma_key_recover.c Moved init of info->pinned_pages to ma_open.c Moved _ma_finalize_row() to maria_key_recover.h Renamed some variables to avoid variable name conflicts Mark page_link.changed for blocks we change directly Simplify handling of undo link when writing LOGREC_UNDO_ROW_INSERT (old code crashed when having redo for index) storage/maria/ma_blockrec.h: Removed extra empty line storage/maria/ma_checkpoint.c: Remove not needed trnman.h storage/maria/ma_close.c: Free pinned pages (which are now always allocated) storage/maria/ma_control_file.c: Give error message if we fail to open control file storage/maria/ma_delete.c: Changes for redo logging (first part, logging of underflow not yet done) - Log undo-key-delete - Log delete of key - Updated arguments to _ma_fetch_keypage(), _ma_dispose(), _ma_write_keypage(), _ma_insert() - Added new arguments to some functions to be able to write redo information - Mark key pages as changed when we write with PAGECACHE_LOCK_LEFT_WRITELOCKED Remove one not needed _ma_write_keypage() in d_search() when upper level will do the write anyway Changed 2 bmove_upp() to bmove() as this made code easer to understand More function comments Indentation fixes storage/maria/ma_ft_update.c: New arguments to _ma_write_keypage() storage/maria/ma_loghandler.c: Fixed some DBUG_PRINT messages Simplify code Added new log entrys for key page redo Renamed some variables to avoid variable name shadowing storage/maria/ma_loghandler.h: Moved some defines here Added define for storing key number on key pages Added new translog record types Added enum for type of operations in LOGREC_REDO_INDEX storage/maria/ma_open.c: Always allocate info.pinned_pages (we need now also for normal key page usage) Update keyinfo->key_nr Added virtual functions to convert record position o number to be stored on key pages Update keyinfo->write_comp_flag to value of search flag to be used when writing key storage/maria/ma_page.c: Added redo for key pages - Extended _ma_fetch_keypage() with type of lock to put on page and address to used MARIA_PINNED_PAGE - _ma_fetch_keypage() now pin's pages if needed - Extended _ma_write_keypage() with type of locks to be used - ma_dispose() now locks info->s->state.key_del from other threads - ma_dispose() writes redo log record - ma_new() locks info->s->state.key_del from other threads if it was used - ma_new() now pins read page Other things: - Removed some not needed arguments from _ma_new() and _ma_dispose) - Added some new variables to simplify code - If EXTRA_DEBUG is used, do crc on full page to catch not unitialized bytes storage/maria/ma_pagecache.h: Applied patch from Sanja to add extra argument to pagecache_unlock_by_ulink() to mark if page was changed Added some defines for pagecache priority levels that one can use storage/maria/ma_range.c: Added new arguments for call to _ma_fetch_keypage() storage/maria/ma_recovery.c: - Added hooks for new translog types: REDO_INDEX, REDO_INDEX_NEW_PAGE, REDO_INDEX_FREE_PAGE, UNDO_KEY_INSERT, UNDO_KEY_DELETE and UNDO_KEY_DELETE_WITH_ROOT. - Moved variable declarations to start of function (portability fixes) - Removed some not needed initializations - Set only relevant state changes for each redo/undo entry storage/maria/lockman.c: Removed end space storage/maria/ma_check.c: Removed end space storage/maria/ma_create.c: Removed end space storage/maria/ma_locking.c: Removed end space storage/maria/ma_packrec.c: Removed end space storage/maria/ma_pagecache.c: Removed end space storage/maria/ma_panic.c: Removed end space storage/maria/ma_rt_index.c: Added new arguments for call to _ma_fetch_keypage(), _ma_write_keypage(), _ma_dispose() and _ma_new() Fixed indentation storage/maria/ma_rt_key.c: Added new arguments for call to _ma_fetch_keypage() storage/maria/ma_rt_split.c: Added new arguments for call to _ma_new() Use new keypage header Added new arguments for call to _ma_write_keypage() storage/maria/ma_search.c: Updated comments & indentation Added new arguments for call to _ma_fetch_keypage() Made some variables and arguments const Added virtual functions for converting row position to number to be stored in key use MARIA_RECORD_POS of record position instead of my_off_t Record in MARIA_KEY_PARAM how page was changed one key insert (needed for REDO) storage/maria/ma_sort.c: Removed end space storage/maria/ma_statrec.c: Updated arguments for call to _ma_rec_pos() storage/maria/ma_test1.c: Fixed too small buffer to init_pagecache() Fixed bug when using insert_count and test_flag storage/maria/ma_test2.c: Use more resonable pagecache size Remove not used code Reset blob_length to fix wrong output message storage/maria/ma_test_all.sh: Fixed wrong test storage/maria/ma_write.c: Lots of new code to handle REDO of key pages No logic changes because of REDO code, mostly adding new arguments and adding new code for logging Added new arguments for calls to _ma_fetch_keypage(), _ma_write_keypage() and similar functions Move setting of comp_flag in ma_ck_wrte_btree() from runtime to maria_open() Zerofill new used pages for: - To remove possible sensitive data left in buffer - To get idenitical data on pages after running redo - Better compression of pages if archived storage/maria/maria_chk.c: Added information if table is crash safe storage/maria/maria_def.h: New virtual function to convert between record position on key and normal record position Aded mutex and extra variables to handle locking of share->state.key_del Moved some structure variables to get things more aligned Added extra arguments to MARIA_KEY_PARAM to be able to remember what was changed on key page on key insert Added argument to MARIA_PINNED_PAGE to indicate if page was changed Updated prototypes for functions Added some structures for signaling changes in REDO handling storage/maria/unittest/ma_pagecache_single.c: Updated arguments for changed function calls storage/myisam/mi_check.c: Made calc_check_checksum virtual storage/myisam/mi_checksum.c: Update checksums to ignore null columns storage/myisam/mi_create.c: Mark if table has null column (to know when we have to use mi_checksum()) storage/myisam/mi_open.c: Added virtual function for calculating checksum to be able to easily ignore NULL fields storage/myisam/mi_test2.c: Fixed bug storage/myisam/myisamdef.h: Added virtual function for calculating checksum during check table Removed ha_key_cmp() as this is in handler.h storage/maria/ma_key_recover.c: New BitKeeper file ``storage/maria/ma_key_recover.c'' storage/maria/ma_key_recover.h: New BitKeeper file ``storage/maria/ma_key_recover.h'' storage/maria/ma_key_redo.c: New BitKeeper file ``storage/maria/ma_key_redo.c''
2007-11-14 18:08:06 +01:00
static inline void _ma_finalize_row(MARIA_HA *info)
{
info->trn->rec_lsn= LSN_IMPOSSIBLE;
}
/* unpinning is often the last operation before finalizing */
static inline void _ma_unpin_all_pages_and_finalize_row(MARIA_HA *info,
LSN undo_lsn)
{
_ma_unpin_all_pages(info, undo_lsn);
_ma_finalize_row(info);
}
extern my_bool _ma_lock_key_del(MARIA_HA *info, my_bool insert_at_end);
extern void _ma_unlock_key_del(MARIA_HA *info);
static inline void _ma_fast_unlock_key_del(MARIA_HA *info)
{
if (info->used_key_del)
_ma_unlock_key_del(info);
}