mirror of
https://github.com/MariaDB/server.git
synced 2025-01-17 12:32:27 +01:00
18 commits
Author | SHA1 | Message | Date | |
---|---|---|---|---|
Monty
|
7def2877e3 |
Write error message if aria_log.??????? files are missing
This can happen if one uses a backup where not all aria_log.* files are copied or if the last one is too short. In this case the data files will contain data that is not in the logs and recovery will fail. Other things: - Fixed tprint() to not print extra new line to debug trace |
||
Michal Schorm
|
17b4f99928 |
Update FSF address
This commit is based on the work of Michal Schorm, rebased on the earliest MariaDB version. Th command line used to generate this diff was: find ./ -type f \ -exec sed -i -e 's/Foundation, Inc., 59 Temple Place, Suite 330, Boston, /Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, /g' {} \; \ -exec sed -i -e 's/Foundation, Inc. 59 Temple Place.* Suite 330, Boston, /Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, /g' {} \; \ -exec sed -i -e 's/MA.*.....-1307.*USA/MA 02110-1335 USA/g' {} \; \ -exec sed -i -e 's/Foundation, Inc., 59 Temple/Foundation, Inc., 51 Franklin/g' {} \; \ -exec sed -i -e 's/Place, Suite 330, Boston, MA.*02111-1307.*USA/Street, Fifth Floor, Boston, MA 02110-1335 USA/g' {} \; \ -exec sed -i -e 's/MA.*.....-1307/MA 02110-1335/g' {} \; |
||
Michael Widenius
|
bf0e1f44c6 |
Fixed portability problem with partiton_error.test
Added option to aria_read_log to crash recovery at certain points in the recovery process. Fixed bug that caused future recovery attempts to fail if we got a crash/got killed during closing of tables at end of recovery process. mysql-test/mysql-test-run.pl: Don't abort if 'var' points to stale place; Just remove it. mysql-test/suite/maria/r/maria.result: Fixed wrong indentation mysql-test/t/partition_error.test: Fixed portability problem with partiton_error.test storage/maria/ma_close.c: More DBUG_PRINT info storage/maria/ma_pagecache.c: Copy flush_log_callback when writing to page cache. This fixes problem in recovery when switching from mode of file storage/maria/ma_recovery.c: Added option to aria_read_log to crash recovery at certain points in the recovery process. storage/maria/ma_recovery.h: Added option to aria_read_log to crash recovery at certain points in the recovery process. storage/maria/maria_chk.c: Align aria_chk -d output Don't write warning Aria table '...' is usable but should be fixed if the table was before marked as crashed but now is ok storage/maria/maria_read_log.c: Added option to aria_read_log to crash recovery at certain points in the recovery process. |
||
Michael Widenius
|
5b0afd8a0d |
Bug fixing in Aria:
- Fixed some bugs in recovery of blobs - Don't ASSERT() on checksum errors when running check table - Added to maria_read_log option --tables-to-redo=list-of-tables to only recover some tables (good for debugging) storage/maria/ma_blockrec.c: Don't ASSERT() on checksum errors when running check table Fixed bug in recovery of blog page that was not in dirty pages list storage/maria/ma_check.c: Don't ASSERT() on checksum errors when running check table storage/maria/ma_recovery.c: Handling of --tables-to-redo storage/maria/ma_recovery.h: Handling of --tables-to-redo storage/maria/ma_recovery_util.c: Give better warning if table was not in dirty pages list storage/maria/maria_def.h: Added in_check_table storage/maria/maria_read_log.c: Added --tables-to-redo=list-of-tables to only recover some tables (good for debugging) Cleaned up message when wrong arguments |
||
Michael Widenius
|
f66d2500c4 |
Remove not used files
Added option: --warning-for-wrong-transaction-id to maria_check mysql-test/r/innodb-use-sys-malloc.result: Remove not used files mysql-test/t/innodb-use-sys-malloc-master.opt: Remove not used files storage/maria/ma_recovery.c: Added support for --lsn-end storage/maria/ma_recovery.h: New prototype storage/maria/ma_write.c: Added logging of KEY_OP_CHECK for all possible keyblock splits. (Helps us find error in log file handling) storage/maria/maria_chk.c: Added option: --warning-for-wrong-transaction-id --update-status now resets open count if check succeds storage/maria/maria_read_log.c: Added option --end-lsn for only appling part of log (good for debugging) Changed options to use '-' instead of '_' |
||
Guilhem Bichot
|
a5bcb63f45 |
WL#4374 "Maria - force start if Recovery fails multiple times"
http://forge.mysql.com/worklog/task.php?id=4374 new option --maria-force-start-after-recovery-failures=N; number of consecutive recovery failures (failures of log reading or recovery processing, anything in [translog_init(),maria_recovery_from_log()]) is stored in the control file; if at a Maria start they are more than N, logs are removed. This is for automated systems which have to run whatever happens. As tables risk staying corrupted, --maria-recover should also be used on them: this revision makes maria-recover work (it was disabled). Fixed bug in translog_is_log_files(). translog_init() now prints message to error log if failed. Removed \0 in the output of SHOW ENGINE MARIA LOGS; removed hard-coded engine name there. KNOWN_BUGS.txt: As option --maria-force-start-after-recovery-failures is added, it corresponds to the wish "we should fix that if this happens etc". LOAD INDEX is not ignored since a few weeks. Listed concurrency bugs have been fixed some time ago. Recovery of fulltext and GIS indexes works since a few weeks. mysql-test/include/maria_make_snapshot.inc: configurable prefix in table's name (so far 't' or 't_corrupted') mysql-test/include/maria_make_snapshot_for_comparison.inc: configurable prefix in table's name (so far 't' or 't_corrupted') mysql-test/include/maria_make_snapshot_for_feeding_recovery.inc: configurable prefix in table's name (so far 't' or 't_corrupted') mysql-test/include/maria_verify_recovery.inc: configurable prefix in table's name (so far 't' or 't_corrupted') mysql-test/lib/mtr_report.pl: new test maria-recover.test generates expected corruption warnings in the error log. maria-recovery.test's corrupted table is renamed to t_corrupted1 instead of t1. mysql-test/r/maria-preload.result: result update. maria_pagecache_read* values are similar to the previous version of this file, though a bit bigger because using the information_schema and the join leads to some internal maria temp table being used, and thus some blocks of it being read. mysql-test/r/maria-purge.result: engine's name in SHOW ENGINE MARIA LOGS changed. mysql-test/r/maria-recover.result: result for new test. We see corruption messages at first SELECT and then none at second SELECT, expected. mysql-test/r/maria-recovery.result: result update mysql-test/r/maria.result: new variables show up mysql-test/t/disabled.def: BUG#34911 is not fixed but the test had been made independent of the bug (workaround). A new bug (crash) has popped recently, so it has to stay disabled (BUG#35107). mysql-test/t/maria-preload.test: Work around BUG#34911 "FLUSH STATUS doesn't flush what it should": compute differences in status variables before and after relevant queries mysql-test/t/maria-recover-master.opt: test --maria-recover mysql-test/t/maria-recover.test: Test of the --maria-recover option (build a corrupted table and see if it is auto-repaired) mysql-test/t/maria-recovery-big.test: update for new API of include/maria*.inc mysql-test/t/maria-recovery-bitmap.test: update for new API of include/maria*.inc mysql-test/t/maria-recovery.test: update for new API of include/maria*.inc. Corrupted table t1 renamed to t_corrupted1, so that mtr_report.pl does not blindly remove all corruption messages for t1 which is a common name. storage/maria/ha_maria.cc: Enabling maria-recover. Adding option and global variable --maria_force_start_after_recovery_failures: ha_maria_init() calls mark_recovery_start() and mark_recovery_success() to keep track of failed consecutive recoveries and remove logs if needed. Removed \0 in the output of SHOW ENGINE MARIA LOGS; removed hard-coded engine name there. storage/maria/ma_checkpoint.c: new prototype storage/maria/ma_control_file.c: Storing in one byte in the control file, the number of consecutive recovery failures. storage/maria/ma_control_file.h: new prototype storage/maria/ma_init.c: new prototype storage/maria/ma_locking.c: Need to update open_count on disk at first write and close for transactional tables, like we already did for non-transactional tables, otherwise we cannot notice that the table is dubious. storage/maria/ma_loghandler.c: translog_is_log_files() is made more generic to serve either to search or to delete logs (the latter is for --maria-force-start-after-recovery-failures). It also had a bug (always returned FALSE). storage/maria/ma_loghandler.h: export function because ha_maria::mark_recovery_start() needs it storage/maria/ma_recovery.c: changing name of maria_recover() to distinguish from the maria-recover option. storage/maria/ma_recovery.h: changing name of maria_recover() to distinguish from the maria-recover option. storage/maria/ma_test_force_start.pl: Test of --maria-force-start-after-recovery-failures (and also, to be realistic, of --maria-recover). This is standalone because mysql-test-run does not support testing that multiple mysqld restarts expectedly failed. I'll have to run it on my machine and also on a Windows machine. storage/maria/unittest/ma_control_file-t.c: adding recovery_failures to the test storage/maria/unittest/ma_test_loghandler_multigroup-t.c: fix for compiler warning (unused variable in non-debug build) |
||
unknown
|
bfd5c273c4 |
UNDO of rows now puts back all part of the row on their original pages and positions
Added variable _dbug_on_ to speed up execution when DBUG is not going to be used Added --debug-on option to mysqld (to be able to turn of DBUG with --debug-on=0) Fixed some bugs with 'non_flushable' marking of bitmap pages Don't use 'non_flushable' marking of bitmap pages for not transactional tables SHOW CREATE TABLE now shows if table was created with page checksums Fixed a lot of bugs with BLOB handling in case of update/REDO and UNDO More tests (especially for blobs) and DBUG_ASSERTS() More readable output from maria_read_log and maria_chk Fixed wrong shift that caused Maria to crash on files > 4G Mark tables as crashed of REDO fails dbug/dbug.c: Changed to use my_bool (allowed me to remove some windows specific code) Added variable _dbug_on_ to speed up execution when DBUG is not going to be used Removed initialization of variables if not needed include/my_dbug.h: Use my_bool for some functions that was defined as BOOLEAN in dbug.c code Added DBUGGER_ON/DEBUGGER_OFF to speed up execution when DBUG is not used include/my_global.h: Define my_bool early Increase MY_HOW_OFTEN_TO_WRITE as computers are now faster than 10 years ago mysql-test/mysql-test-run.pl: Added debug-on=0 to speed up tests mysql-test/r/maria-recovery.result: Added new test by Guilhem to test if UNDO_ROW_DELETE preserves rowid mysql-test/r/maria.result: Added testing of page checksums mysql-test/t/crash_commit_before-master.opt: Added --debug-on as test require DBUG to work mysql-test/t/maria-recovery-bitmap-master.opt: Added --debug-on as test require DBUG to work mysql-test/t/maria-recovery-master.opt: Added --debug-on as test require DBUG to work mysql-test/t/maria-recovery.test: Added new test by Guilhem to test if UNDO_ROW_DELETE preserves rowid mysql-test/t/maria.test: Added testing of page checksums sql/mysqld.cc: Added --debug-on option (to be able to turn of DBUG with --debug-on=0) Indentation fixes Removed end spaces sql/sql_show.cc: Allow update_create_info() to inform MySQL if PACK_KEYS, NO_PACK_KEYS, CHECKSUM, PAGE_CHECKSUM or DELAY_KEY_WRITE is used storage/maria/Makefile.am: Added ma_test_big.sh storage/maria/ha_maria.cc: Store in create_info if page checksums are used (For SHOW CREATE TABLE) storage/maria/ma_bitmap.c: Added _ma_bitmap_wait_or_flush() to cause reader of bitmap pages to wait with reading until bitmap is flushed. Use TAIL_PAGE_COUNT_MARKER for tail pages Set 'sub_blocks' for and only for the head page or for the first extent of a blob. This is needed for store_extent_info() to be able to set START_EXTENT_BIT's Don't allocate more than 0x3ffff pages in one extent (We need bit 0x4000 as a START_EXTENT_BIT) Increase the calculated 'head_length' with the number of bytes used for extents. Update row->space_on_head_page also in _ma_bitmap_find_new_place() Make _ma_bitmap_get_page_bits() global. (Needed for UNDO handling) Changed _ma_bitmap_flushable() to take MARIA_HA instead of MARIA_SHARE. This was needed to be able to mark the handler if we had a 'non_flushable' call pending or not. Don't use 'non_flushable' marking of bitmap pages for not transactional tables. Added BLOCKUSED_USE_ORG_BITMAP handling also for tail pages. Added more DBUG_ASSERT() to find possible errors in other code Some code simplications by adding new local variables storage/maria/ma_blockrec.c: UNDO of rows now puts back all part of the row on their original pages and positions. Changed UNDO of DELETE and UNDO of UPDATE to contain information about the original length of data on head block and also extent information This changes a lot of logic as now an insert of a row on a page may happen to any position (and not just to the first or next free) Use PAGE_COUNT to mark if an extent is the start of of a blob. (Needed for extent_to_bitmap_blocks()) Added check_directory() for checking that directroy entries are correct. Added checking of row checksums when reading rows (with EXTRA_DEBUG) Added make_space_for_directory() and extend_directory() for doing expansion of directory Added get_rowpos_in_head_or_tail_page() to be able to store head/tail on original position in UNDO Added extent_to_bitmap_blocks() to be able to generate original bitmap blocks from UNDO entry Added _ma_update_at_original_place() for UNDO of DELETES Added row->min_length to hold minmum required space needed on head page Changed find_free_position() to use make_space_for_directory() Changed make_empty_page() to allow optional creation of directory entry Changed delete_head_or_tail() and _ma_apply_undo_row_isnert() to not copy pagecache block (speed optimization) Changed _ma_apply_redo_insert_row_head_or_tail() to be able to insert new row at any position on 'new' page Changed _ma_apply_undo_row_delete() and _ma_apply_undo_row_update() to put row in it's original position Ensure allocation of tail blocks are of at least MIN_TAIL_SIZE. Ensure we store pages in pinned pages even if read failed. (If not we will have pages pinned forever in page cache) Write original extent information in UNDO entry, not compacted ones (we need position to tails!) When setting BLOCKUSED_USED, don't clear other bits (we have to preserve BLOCKUSED_USE_ORG_BITMAP) Fixed som bugs in directory handling Fixed bug where we wrote wrong lsn to blob pages Added separate blob_buffer for fixing bug when updating row that had char/varchar that spanned several pages and also had blobs Ensure we call _ma_bitmap_flushable() also in case of errors When doing an update, first delete old entries, then search in bitmap for where to put new information Info->s -> share Rowid -> rowid More DBUG_ASSERT() storage/maria/ma_blockrec.h: Added START_EXTENT_BIT and TAIL_PAGE_COUNT_MARKER Added _ma_bitmap_wait_or_flush() and _ma_bitmap_get_page_bits() storage/maria/ma_check.c: Don't write extra empty line if there is no deleted blocks Ignore START_EXTENT_BIT's in page count Call _ma_fast_unlock_key_del() to free key_del link storage/maria/ma_close.c: Ensure that used_key_del is 0. (If not, someone forgot to call _ma_unlock_key_del()) storage/maria/ma_create.c: Changed constant to macro storage/maria/ma_delete.c: For deleted keys, log also position to row storage/maria/ma_extra.c: Release blob buffer at maria_reset() if bigger than MARIA_SMALL_BLOB_BUFFER storage/maria/ma_key_recover.c: Added bzero() of LSN that confused paged cache in case of uninitialized block Mark file crashed if applying of index changes fails Added calls to _ma_fast_unlock_key_del() for protection of shared key_del link. storage/maria/ma_locking.c: Added usage of MARIA_FILE_OPEN_COUNT_OFFSET Added _ma_mark_file_crashed() storage/maria/ma_loghandler.c: Fixed bug where we logged uninitialized memory storage/maria/ma_open.c: Moved state->changed to be at start of state info on disk to allow one to easly mark files as crashed storage/maria/ma_page.c: Disable 'dummy' checksumming of pages as this gave false warnings. (Need to investigate if this is ever needed) storage/maria/ma_pagecache.c: Fixed wrong shift that caused Maria to crash on files > 4G storage/maria/ma_recovery.c: In case of errors, start writing on new line if we where in %## %## printing mode (Made errors more readable) Changed global variable name from warnings -> recovery_warnings Use MARIA_FILE_CREATE_RENAME_LSN_OFFSET instead of constant Removed special handling of row position for deleted keys. Keys now always includes row positions _ma_apply_undo_row_delete() now gets page and row position Added check that we don't loop forever when handling undo's (in case of bug in undo chain) Print name of failed REDO/UNDO storage/maria/ma_recovery.h: Removed old comment storage/maria/ma_static.c: Chaned version number of Maria files to not accidently use old ones (becasue of change of ordering of status variables) storage/maria/ma_test2.c: Added option -u to specify number of rows to update Changed old option -u to be -A, as for ma_test1 Fixed bug in update of rows with blobs (before blobs was always reset to empty on update) First created blob is now of max blob length to ensure we have at least one big blob in the table storage/maria/ma_test_all.sh: More tests storage/maria/ma_test_recovery.expected: Updated results storage/maria/ma_test_recovery: Changed tests to use bigger blobs (not just 1K) Added new tests that tests recovery of update with blobs Removed comparision of .MAD file as it's not guranteed that recovery from scratch gives identical data file as original update (compact_page() may be called at different times during normal execution and during REDO) storage/maria/ma_update.c: Simplify code (changed * to if) storage/maria/maria_chk.c: Make output more readable storage/maria/maria_def.h: Changed 'changed' to int to prepare for more bits Added 2 more bytes to status information Added 'st_mara_row->min_length' for storing min length needed on head page Added 'st_mara_handler->blob_buff & blob_buff_size' for storing blobs Moved all tunning parameters into one block Added MARIA_SMALL_BLOB_BUFFER Added _ma_mark_file_crashed() storage/myisam/mi_test2.c: Fixed bug in update of rows with blobs (before blobs was always reset to empty on update) storage/maria/ma_test_big.sh: Testing of insert, update, delete, recovery and undo of rows with blobs Thanks to the random-ness of ma_test2 this is likely to find most bugs in the row handling |
||
unknown
|
1bc5e3b971 |
WL#3072 - Maria recovery
maria_read_log used to always print a warning message at startup to say it is unsafe if ALTER TABLE was used. Now it prints it only if the log does show the problem (=ALTER TABLE or CREATE SELECT, which both disable logging of REDO_INSERT*). For that, when ha_maria::external_lock() disables transactionality it writes a LOGREC_INCOMPLETE_LOG to the log, which "maria_read_log -a" picks up to write a warning. REPAIR TABLE also disables those REDO_INSERT* but as maria_read_log executes LOGREC_REDO_REPAIR no warning is needed. storage/maria/ha_maria.cc: as we now log a record when disabling transactionility, we need the TRN to be set up first storage/maria/ma_blockrec.c: comment storage/maria/ma_loghandler.c: new type of log record storage/maria/ma_loghandler.h: new type of log record storage/maria/ma_recovery.c: * maria_apply_log() now returns a count of warnings. What currently produces warnings is: - skipping applying UNDOs though there are some (=> inconsistent table) - replaying log (in maria_read_log) though the log contains some ALTER TABLE or CREATE SELECT (log misses REDO_INSERT* for those and is so incomplete). Count of warnings affects the final message of maria_read_log and recovery (though in recovery none of the two conditions above should happen). * maria_read_log used to always print a warning message at startup to say it is unsafe if ALTER TABLE was used. Now it prints it only if the log does show the problem, i.e. ALTER TABLE or CREATE SELECT was used (both disable logging of REDO_INSERT* as those records are not needed for recovery; those missing records in turn make recreation-from-scratch, via maria_read_log, impossible). For that, when ha_maria::external_lock() disables transactionality, _ma_tmp_disable_logging_for_table() writes a LOGREC_INCOMPLETE_LOG to the log, which maria_apply_log() picks up to write a warning. storage/maria/ma_recovery.h: maria_apply_log() returns a count of warnings storage/maria/maria_def.h: _ma_tmp_disable_logging_for_table() grows so becomes a function storage/maria/maria_read_log.c: maria_apply_log can now return a count of warnings, to temper the "SUCCESS" message printed in the end by maria_read_log. Advise users to make a backup first. |
||
unknown
|
08b7152893 |
WL#3072 Maria recovery
- fixes (in recovery, checkpoint, log handler) of bugs found during testing. - new option --check for maria_read_log: with --only-display (which only reads the header) it reads the full record, for debugging. storage/maria/ma_loghandler.c: importing patch from Sanja for bug of translog_next_LSN() found during recovery storage/maria/ma_loghandler_lsn.h: better types (0L is 4 bytes on some platforms, it causes problems when used into lsn_store(): right shift >= width of type. storage/maria/ma_pagecache.c: work around infamous "PAGECACHE_PLAIN_PAGE used for transactional tables in specialm case"; REDO phase disables logging and this causes pages to be PAGECACHE_PLAIN_PAGE, thus ignored wrongly by the checkpoint taken at the end of the REDO phase. storage/maria/ma_recovery.c: - a #ifdef which broke maria_read_log in non-debug builds (no output!) - support for maria_read_log --check - detect record corruption before opening the table - updating is_of_horizon requires writing the state - fix for wrong parsing of checkpoint record by recovery storage/maria/ma_recovery.h: support for maria_read_log --check storage/maria/maria_read_log.c: Option --check: --only-display only looks at the header; adding --check tries a translog_read_record() to see if record can be fully read (this is to find bugs). |
||
unknown
|
63ff9877a5 |
WL#3072 Maria recovery
Misc changes: - fix for benign Valgrind error, compiler warnings - fix for a segfault in execution of maria_delete_all_rows() and one when taking multiple checkpoints - fix for too paranoid assertion - adding ability to take checkpoints at the end of the REDO phase and at the end of recovery. - other minor changes storage/maria/ha_maria.cc: The checkpoint done after Recovery is finished, is moved to maria_recover(). storage/maria/ma_bitmap.c: fix for Valgrind error: the "shadow debug copy" of the bitmap page started unitialized and so ma_print_bitmap() would use it uninitialized storage/maria/ma_checkpoint.c: * reset pointers to NULL after freeing them, or we segfault at next checkpoint in my_realloc(). * fix for compiler warnings. storage/maria/ma_delete_all.c: info->trn is NULL for non-transactional tables storage/maria/ma_locking.c: correct assertion (it fired wrongly in execution of REDO_DROP_TABLE due to the maria_extra(HA_PREPARE_FOR_DROP)->_ma_decrement_open_count() ->maria_lock_database(F_UNLCK); another solution would have been to not call _ma_decrement_open_count() (it's ok to have a wrong open count in a table which we are dropping), but the same problem would still exist for REDO_RENAME_TABLE. storage/maria/ma_loghandler.c: fail early if UNRECOVERABLE_ERROR storage/maria/ma_recovery.c: * new argument to maria_apply_log(): should it take checkpoints (at end of REDO phase and at the very end) or no. * moving the call to translog_next_LSN() into parse_checkpoint_record() ("hide the details"). * Refining an error detection for something which could happen if there is a checkpoint record in the log. * Using close_one_table() instead of maria_extra(HA_EXTRA_PREPARE_FOR_DROP|RENAME), as it looks safer, and also changing how close_one_table() works: it now limits itself to scanning all_tables[], thus having one loopp instead of two, which should be faster (as a result, it does not close tables not registered in this array, which is ok as there should not be any). storage/maria/ma_recovery.h: new parameter storage/maria/maria_read_log.c: update to new prototype |
||
unknown
|
95420b947e |
fix for non-debug compilation errors.
Note that non-debug build fails in log handler functions, mail sent. storage/maria/ma_blockrec.c: fix for compiler warning storage/maria/ma_checkpoint.c: Debug build does not catch this situation static int f(); ... f(2); ... static int f(int a, int b); Maybe this is because it believes the declaration is K&R. Non-debug build catches it. Adding (void) as an habit to avoid such errors. storage/maria/ma_checkpoint.h: adding (void) storage/maria/ma_recovery.c: adding (void) storage/maria/ma_recovery.h: adding (void) |
||
unknown
|
9c2ff270fa |
WL#3072 Maria Recovery
* recovery from ha_maria now skips replaying DDLs (too dangerous) * maria_read_log still replays DDLs, print warning about issues * fixes to replaying of REDO_RENAME * don't replay DDLs on corrupted tables (safer) * print a one-line message when really doing a recovery (applies to ha_maria, not maria_read_log) i.e. some REDOs or UNDOs are read. storage/maria/ma_checkpoint.c: fix for assertion failure storage/maria/ma_recovery.c: * Recovery from ha_maria now skips replaying DDLs (as the initial plan said) as this is unsafe in case of crashes during the DDL; applying the records may do harm (destroy important files) so we prefer to leave the "mess" of files untouched. A proper recovery of DDLs requires very careful thinking, probably testing separately the existence of the data and index file instead of using maria_open() which tests the existence of both, and maybe storing create_rename_lsn in the data file too. * maria_read_log still replays DDLs, we print a warning about dangers (due to ALTER TABLE not logging insertions into the tmp table; we will maybe need an option to have logging of those insertions). * fixes to replaying of REDO_RENAME (test create_rename_lsn of 'new_name' table if it exists; if that table exists and is more recent than the record, remove the 'old_name' table). * don't replay DDLs on corrupted tables (play safe) * fail also in non-debug builds if table is open when it should not be (when creating it for example, it should not be already open). * when the trace file is not stdout (i.e. when this is ha_maria), if really doing a recovery (reading REDOs or UNDOs), print a one-line message to stderr to inform about start and end of recovery (useful to know what mysqld is doing, especially if it takes long or crashes). storage/maria/ma_recovery.h: parameter to replay DDLs or not storage/maria/maria_read_log.c: replay DDLs in maria_read_log, to be able to recreate tables from scratch. |
||
unknown
|
e27890cab0 |
WL#3072 Maria recovery
* create page cache before initializing engine and not after, because Maria's recovery needs a page cache * make the creation of a bitmap page more crash-resistent * bugfix (see ma_blockrec.c) * back to old way: create an 8k bitmap page when creating table * preparations for the UNDO phase: recreate TRNs * preparations for Checkpoint: list of dirty pages, testing of rec_lsn to know if page should be skipped during Recovery (unused in this patch as no Checkpoint module pushed yet) * maria_chk tags repaired table with a special LSN * reworking all around in ma_recovery.c (less duplication) mysys/my_realloc.c: noted an issue in my_realloc() sql/mysqld.cc: page cache needs to be created before engines are initialized, because Maria's initialization may do a recovery which needs the page cache. storage/maria/ha_maria.cc: update to new prototype storage/maria/ma_bitmap.c: when creating the first bitmap page we used chsize to 8192 bytes then pwrite (overwrite) the last 2 bytes (8191-8192). If crash between the two operations, this leaves a bitmap page full without its end marker. A later recovery may try to read this page and find it exists and misses a marker and conclude it's corrupted and fail. Changing the chsize to only 8190 bytes: recovery will then find the page is too short and recreate it entirely. storage/maria/ma_blockrec.c: Fix for a bug: when executing a REDO, if the data page is created, data_file_length was increased before _ma_bitmap_set(): _ma_bitmap_set() called _ma_read_bitmap_page() which, due to the increased data_file_length, expected to find a bitmap page on disk with a correct end marker; if the bitmap page didn't exist already in fact, this failed. Fixed by increasing data_file_length only after _ma_read_bitmap_page() has created the new bitmap page correctly. This bug could happen every time a REDO is about creating a new bitmap page. storage/maria/ma_check.c: empty data file has a bitmap page storage/maria/ma_control_file.c: useless parameter to ma_control_file_create_or_open(), just test if this is recovery. storage/maria/ma_control_file.h: new prototype storage/maria/ma_create.c: Back to how it was before: maria_create() creates an 8k bitmap page. Thus (bugfix) data_file_length needs to reflect this instead of being 0. storage/maria/ma_loghandler.c: as ma_test1 and ma_test2 now use real transactions and not dummy_transaction_object, REDO for INSERT/UPDATE/DELETE are always about real transactions, can assert this. A function for Recovery to assign a short id to a table. storage/maria/ma_loghandler.h: new function storage/maria/ma_loghandler_lsn.h: maria_chk tags repaired tables with this LSN storage/maria/ma_open.c: * enforce that DMLs on transactional tables use real transactions and not dummy_transaction_object. * test if table was repaired with maria_chk (which has to been seen as an import of an external table into the server), test validity of create_rename_lsn (header corruption detection) * comments. storage/maria/ma_recovery.c: * preparations for the UNDO phase: recreate TRNs * preparations for Checkpoint: list of dirty pages, testing of rec_lsn to know if page should be skipped during Recovery (unused in this patch as no Checkpoint module pushed yet) * reworking all around (less duplication) storage/maria/ma_recovery.h: a parameter to say if the UNDO phase should be skipped storage/maria/maria_chk.c: tag repaired tables with a special LSN storage/maria/maria_read_log.c: * update to new prototype * no UNDO phase in maria_read_log for now storage/maria/trnman.c: * a function for Recovery to create a transaction (TRN), needed in the UNDO phase * a function for Recovery to grab an existing transaction, needed in the UNDO phase (rollback all existing transactions) storage/maria/trnman_public.h: new functions |
||
unknown
|
6bbca54d7d |
WL#3072 - Maria recovery
Unit test for recovery: runs ma_test1 and ma_test2 (both only with INSERTs and DELETEs; UPDATEs disabled as not handled by recovery) then moves the tables elswhere; recreates tables from the log, and compares and fails if there is a difference. Passes now. Most of maria_read_log.c moved to ma_recovery.c, as it will be re-used for recovery-from-ha_maria. Bugfixes of applying of REDO_INSERT, REDO_PURGE_ROW. Applying of REDO_PURGE_BLOCKS, REDO_DELETE_ALL, REDO_DROP_TABLE, UNDO_ROW_INSERT (in REDO phase only, i.e. just doing records++), UNDO_ROW_DELETE, UNDO_ROW_PURGE. Code cleanups. Monty: please look for "QQ". Sanja: please look for "Sanja". Future tasks: recovery of the bitmap (easy), recovery of the state (make it idempotent), more REDOs (Monty to work on REDO_UPDATE?), UNDO phase... Pushing this cset as it looks safe, contains test and bugfixes which will help Monty implement applying of REDO_UPDATE. sql/handler.cc: typo storage/maria/Makefile.am: Adding ma_test_recovery (which ma_test_all invokes, and which can also be run alone). Most of maria_read_log.c moved to ma_recovery.c storage/maria/ha_maria.cc: comments storage/maria/ma_bitmap.c: fixing comments. 2 -> sizeof(maria_bitmap_marker). Bitmap-related part of _ma_initialize_datafile() moves in bitmap module. Now putting the "bm" signature when creating the first bitmap page (it used to happen only at next open, but that caused an annoying difference when testing Recovery if the original run didn't open the table, and it looks more logical like this: it goes to disk only with its signature correct); see the "QQ" comment towards the _ma_initialize_data_file() call in ma_create.c for more). When reading a bitmap page, verify its signature (happens when normally using the table or when CHECKing it; not when REPAIRing it). storage/maria/ma_blockrec.c: * no need to sync the data file if table is not transactional * Comments, code cleanup (log-related data moved to log-related code block, int5store->page_store). * Store the table's short id into LOGREC_UNDO_ROW_PURGE, like we do for other records (though this record will soon be replaced with a CLR). * If "page" is 1 it means the page which extends from byte page*block_size+1 to (page+1)*block_size (byte number 1 being the first byte of the file). The last byte of the file is data_file_length (same convention). A new page needs to be created if the last byte of the page is beyond the last byte of the file, i.e. (page+1)*block_size+1 > data_file_length, so we correct the test (bug found when testing log applying for ma_test1 -M -T --skip-update). * update the page's LSN when removing a row from it during execution of a REDO_PURGE_ROW record (bug found when testing log applying for ma_test1 -M -T --skip-update). * applying of REDO_PURGE_BLOCKs (limited to a one-page range for now). storage/maria/ma_blockrec.h: new functions. maria_bitmap_marker does not need to be exported. storage/maria/ma_close.c: we can always flush the table's state when closing the last instance of the table. And it is needed for maria_read_log (as it does not use maria_lock_database()). storage/maria/ma_control_file.c: when in Recovery, some assertions should not be used. storage/maria/ma_control_file.h: double-inclusion safe storage/maria/ma_create.c: during recovery, don't log records. Comments. Moving the creation of the first bitmap page to ma_bitmap.c storage/maria/ma_delete_table.c: during recovery, don't log records. Log the end-zero of the dropped table's name, so that recovery can use the string in place without extending it to fit an end zero. storage/maria/ma_loghandler.c: * inwrite_rec_hook also needs access to the MARIA_SHARE, like prewrite_rec_hook. This will be needed to update share->records_diff (in the upcoming patch "recovery of the state"). * LOG_DESC::record_ends_group changed to an enum. * LOG_DESC for LOGREC_REDO_PURGE_BLOCKS and LOGREC_UNDO_ROW_PURGE corrected * Sanja please see the @todo LOG BUG * avoiding DBUG_RETURN(func()) as it gives confusing debug traces. storage/maria/ma_loghandler.h: - log write hooks called while the log's lock is held (inwrite_rec_hook) now need the MARIA_SHARE, like prewrite_rec_hook already had - instead of a bool saying if this record's type ends groups or not, we refine: it may not end a group, it may end a group, or it may be a group in itself. Imagine that we had a physical write failure to a table before we log the UNDO, we still end up in external_lock(F_UNLCK) and then we log a COMMIT: we don't want to consider this COMMIT as ending the group of REDOs (don't want to execute those REDOs during Recovery), that's why we say "COMMIT is a group in itself, it aborts any previous group". This also gives one more sanity check in maria_read_log. storage/maria/ma_recovery.c: New Recovery code, replacing the old pseudocode. Most of maria_read_log moved here. Call-able from ha_maria, but not enabled yet. Compared to the previous version of maria_read_log, some bugs have been fixed, debugging output can go to stdout or a disk file (for now it's useful for me, later it can be changed), execution of REDO_DROP_TABLE, REDO_DELETE_ALL, REDO_PURGE_BLOCKS has been added. Duplicate code has been factored into functions. We abort an unfinished group of records if we see a record which is a group in itself (like COMMIT). No need for maria_panic() after a bug (which caused tables to not be closed) was fixed; if there is yet another bug I prefer to see it. When opening a table for Recovery, set data_file_length and key_file_length to their real physical value (these are the easiest state members to restore :). Warn us if the last page was truncated (but Recovery handles it). MARIA_SHARE::state::state::records is now partly recovered (not idempotent, but works if recreating tables from scracth). When applying a REDO to a page, stamp it with the UNDO's LSN (current_group_end_lsn), not with the REDO's LSN; it makes the table more identical to the original table (easier to compare the two tables in the end). Big thing missing: some types of REDOs are not handled, and the UNDO phase does not exist (missing functions to execute UNDOs to actually rollback). So for now tests are only inserting/deleting a few 100 rows, closing the table and seeing if the log is applied ok; it works. UPDATE not handled. storage/maria/ma_recovery.h: new functions: ma_recover() for recovery from inside ha_maria; _ma_apply_log() for maria_read_log (ma_recover() calls _ma_apply_log()). Btw, we need to not use the word "recover" for REPAIR/maria_chk anymore. storage/maria/ma_rename.c: don't write log records during recovery storage/maria/ma_test2.c: - fail if maria_info() or other subtests find some wrong information - new option -g to skip updates. - init the translog before creating the table, so that log applying can work. - in "#if 0" you'll see some fixed bugs (will be removed). storage/maria/ma_test_all.sh: cleanup files. Test log applying. storage/maria/maria_read_log.c: most of the logic moves to ma_recovery.c to be shared between maria_read_log and recovery-from-inside-mysqld. See ma_recovery.c for additional changes made to the moved code. storage/maria/ma_test_recovery: unit test for Recovery. Tests insert and delete, REDO_UPDATE not yet coded. Script is called from ma_test_all. Can run standalone. |
||
unknown
|
adac9798bf |
WL#3072 Maria Recovery
- new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that. |
||
unknown
|
46922b5125 |
GPL license update (same change as was done for all files in 5.1).
storage/maria/Makefile.am: GPL license update storage/maria/ft_maria.c: GPL license update storage/maria/ha_maria.cc: GPL license update storage/maria/ha_maria.h: GPL license update storage/maria/lockman.c: GPL license update storage/maria/lockman.h: GPL license update storage/maria/ma_bitmap.c: GPL license update storage/maria/ma_blockrec.c: GPL license update storage/maria/ma_blockrec.h: GPL license update storage/maria/ma_cache.c: GPL license update storage/maria/ma_changed.c: GPL license update storage/maria/ma_check.c: GPL license update storage/maria/ma_checkpoint.c: GPL license update storage/maria/ma_checkpoint.h: GPL license update storage/maria/ma_checksum.c: GPL license update storage/maria/ma_close.c: GPL license update storage/maria/ma_control_file.c: GPL license update storage/maria/ma_control_file.h: GPL license update storage/maria/ma_create.c: GPL license update storage/maria/ma_dbug.c: GPL license update storage/maria/ma_delete.c: GPL license update storage/maria/ma_delete_all.c: GPL license update storage/maria/ma_delete_table.c: GPL license update storage/maria/ma_dynrec.c: GPL license update storage/maria/ma_extra.c: GPL license update storage/maria/ma_ft_boolean_search.c: GPL license update storage/maria/ma_ft_eval.c: GPL license update storage/maria/ma_ft_eval.h: GPL license update storage/maria/ma_ft_nlq_search.c: GPL license update storage/maria/ma_ft_parser.c: GPL license update storage/maria/ma_ft_stem.c: GPL license update storage/maria/ma_ft_test1.c: GPL license update storage/maria/ma_ft_test1.h: GPL license update storage/maria/ma_ft_update.c: GPL license update storage/maria/ma_ftdefs.h: GPL license update storage/maria/ma_fulltext.h: GPL license update storage/maria/ma_info.c: GPL license update storage/maria/ma_init.c: GPL license update storage/maria/ma_key.c: GPL license update storage/maria/ma_keycache.c: GPL license update storage/maria/ma_least_recently_dirtied.c: GPL license update storage/maria/ma_least_recently_dirtied.h: GPL license update storage/maria/ma_locking.c: GPL license update storage/maria/ma_open.c: GPL license update storage/maria/ma_packrec.c: GPL license update storage/maria/ma_page.c: GPL license update storage/maria/ma_panic.c: GPL license update storage/maria/ma_preload.c: GPL license update storage/maria/ma_range.c: GPL license update storage/maria/ma_recovery.c: GPL license update storage/maria/ma_recovery.h: GPL license update storage/maria/ma_rename.c: GPL license update storage/maria/ma_rfirst.c: GPL license update storage/maria/ma_rkey.c: GPL license update storage/maria/ma_rlast.c: GPL license update storage/maria/ma_rnext.c: GPL license update storage/maria/ma_rnext_same.c: GPL license update storage/maria/ma_rprev.c: GPL license update storage/maria/ma_rrnd.c: GPL license update storage/maria/ma_rsame.c: GPL license update storage/maria/ma_rsamepos.c: GPL license update storage/maria/ma_rt_index.c: GPL license update storage/maria/ma_rt_index.h: GPL license update storage/maria/ma_rt_key.c: GPL license update storage/maria/ma_rt_key.h: GPL license update storage/maria/ma_rt_mbr.c: GPL license update storage/maria/ma_rt_mbr.h: GPL license update storage/maria/ma_rt_split.c: GPL license update storage/maria/ma_rt_test.c: GPL license update storage/maria/ma_scan.c: GPL license update storage/maria/ma_search.c: GPL license update storage/maria/ma_sort.c: GPL license update storage/maria/ma_sp_defs.h: GPL license update storage/maria/ma_sp_key.c: GPL license update storage/maria/ma_sp_test.c: GPL license update storage/maria/ma_static.c: GPL license update storage/maria/ma_statrec.c: GPL license update storage/maria/ma_test1.c: GPL license update storage/maria/ma_test2.c: GPL license update storage/maria/ma_test3.c: GPL license update storage/maria/ma_unique.c: GPL license update storage/maria/ma_update.c: GPL license update storage/maria/ma_write.c: GPL license update storage/maria/maria_chk.c: GPL license update storage/maria/maria_def.h: GPL license update storage/maria/maria_ftdump.c: GPL license update storage/maria/maria_pack.c: GPL license update storage/maria/tablockman.c: GPL license update storage/maria/tablockman.h: GPL license update storage/maria/trnman.c: GPL license update storage/maria/trnman.h: GPL license update |
||
unknown
|
cdf831cf94 |
WL#3071 Maria checkpoint:
changing pseudocode to use the structures of the Maria pagecache ("pagecache->changed_blocks" etc) and other Maria structures inherited from MyISAM (THR_LOCK_maria etc). mysys/mf_pagecache.c: comment storage/maria/ma_checkpoint.c: changing pseudocode to use the structures of the Maria pagecache ("pagecache->changed_blocks" etc) and other Maria structures inherited from MyISAM (THR_LOCK_maria etc). storage/maria/ma_checkpoint.h: copyright storage/maria/ma_control_file.c: copyright storage/maria/ma_control_file.h: copyright storage/maria/ma_least_recently_dirtied.c: copyright storage/maria/ma_least_recently_dirtied.h: copyright storage/maria/ma_recovery.c: copyright storage/maria/ma_recovery.h: copyright storage/maria/unittest/Makefile.am: copyright |
||
unknown
|
a1f25544d5 |
WL#3234 "Maria - control file manager"
- fixes to the control file module - unit test for it - renames of all Maria files I created to start with ma_ storage/maria/ma_checkpoint.c: Rename: storage/maria/checkpoint.c -> storage/maria/ma_checkpoint.c storage/maria/ma_checkpoint.h: Rename: storage/maria/checkpoint.h -> storage/maria/ma_checkpoint.h storage/maria/ma_least_recently_dirtied.c: Rename: storage/maria/least_recently_dirtied.c -> storage/maria/ma_least_recently_dirtied.c storage/maria/ma_least_recently_dirtied.h: Rename: storage/maria/least_recently_dirtied.h -> storage/maria/ma_least_recently_dirtied.h storage/maria/ma_recovery.c: Rename: storage/maria/recovery.c -> storage/maria/ma_recovery.c storage/maria/ma_recovery.h: Rename: storage/maria/recovery.h -> storage/maria/ma_recovery.h storage/maria/Makefile.am: control file module and its unit test program storage/maria/ma_control_file.c: DBUG_ tags. Fix for gcc warnings. log_no -> logno (I felt "_no" sounded like a standalone "No" word). ma_ prefix for some functions. last_checkpoint_lsn_at_startup -> last_checkpoint_lsn (no need to make special vars for the values at startup). Same for last_logno. ma_control_file_write_and_force() now updates last_checkpoint_lsn and last_logno, the idea being that they belong to the module, others should not update them. And thus when the module shuts down, it zeroes those vars. storage/maria/ma_control_file.h: importing structs from Sanja to get the control file module to compile; we'll remove that when Sanja pushes the log handler. CONTROL_FILE_IMPOSSIBLE_LOGNO is 0, not FFFFFFFF. storage/maria/ma_control_file_test.c: Unit test program for the Maria control file module. Modelled after other ma_test* files in this directory (so, does not follow the unit test framework recently introduced with libtap; TODO as a task on all ma_test* programs). We test that writing to the control file works, and re-reading from it too, we check (by reading the file by ourselves) that its content on disk is correct, and check that a corrupted control file is detected. |
Renamed from storage/maria/recovery.h (Browse further)