mirror of
https://github.com/MariaDB/server.git
synced 2025-01-16 20:12:31 +01:00
34 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Michael Widenius
|
e6cf286b5d |
Fixed LP#605798 RQG: Table corruption after Maria engine recovery - "Wrong data in bitmap"
maria_chk & maria_read_log now reads block size from control file. mysql-test/suite/maria/r/maria.result: Updated results after trivial change of maria_chk's output storage/maria/ma_bitmap.c: More DBUG_PRINT storage/maria/ma_blockrec.c: Fixed bug that we didn't mark page full in bitmap if directory is full storage/maria/ma_check.c: Write out if directory is full for errors in bitmap storage/maria/ma_control_file.c: Don't give error for wrong block size if block size is 0 storage/maria/maria_chk.c: Read block size from control file In case of -dvv, write also out bitmap information (good for debugging) storage/maria/maria_read_log.c: Read block size from control file Fixed that maria_read_log works with different page size than TRANSLOG_PAGE_SIZE |
||
|
Michael Widenius
|
e4febda89e |
Fixed that packed transid can be compared byte per byte and we get them in increasing order
The old way to store the length prefix was (256 - length); This is now changed to (length -249) Fixed also that some defines to have a MARIA_ prefix storage/maria/ma_control_file.c: Added comment storage/maria/ma_key.c: Added MARIA_ prefix to some defines Changed how packed transid length was stored storage/maria/ma_open.c: Added MARIA_ prefix to some defines storage/maria/maria_def.h: Added MARIA_ prefix to some defines Changed how packed transid length was stored |
||
|
Guilhem Bichot
|
a5bcb63f45 |
WL#4374 "Maria - force start if Recovery fails multiple times"
http://forge.mysql.com/worklog/task.php?id=4374 new option --maria-force-start-after-recovery-failures=N; number of consecutive recovery failures (failures of log reading or recovery processing, anything in [translog_init(),maria_recovery_from_log()]) is stored in the control file; if at a Maria start they are more than N, logs are removed. This is for automated systems which have to run whatever happens. As tables risk staying corrupted, --maria-recover should also be used on them: this revision makes maria-recover work (it was disabled). Fixed bug in translog_is_log_files(). translog_init() now prints message to error log if failed. Removed \0 in the output of SHOW ENGINE MARIA LOGS; removed hard-coded engine name there. KNOWN_BUGS.txt: As option --maria-force-start-after-recovery-failures is added, it corresponds to the wish "we should fix that if this happens etc". LOAD INDEX is not ignored since a few weeks. Listed concurrency bugs have been fixed some time ago. Recovery of fulltext and GIS indexes works since a few weeks. mysql-test/include/maria_make_snapshot.inc: configurable prefix in table's name (so far 't' or 't_corrupted') mysql-test/include/maria_make_snapshot_for_comparison.inc: configurable prefix in table's name (so far 't' or 't_corrupted') mysql-test/include/maria_make_snapshot_for_feeding_recovery.inc: configurable prefix in table's name (so far 't' or 't_corrupted') mysql-test/include/maria_verify_recovery.inc: configurable prefix in table's name (so far 't' or 't_corrupted') mysql-test/lib/mtr_report.pl: new test maria-recover.test generates expected corruption warnings in the error log. maria-recovery.test's corrupted table is renamed to t_corrupted1 instead of t1. mysql-test/r/maria-preload.result: result update. maria_pagecache_read* values are similar to the previous version of this file, though a bit bigger because using the information_schema and the join leads to some internal maria temp table being used, and thus some blocks of it being read. mysql-test/r/maria-purge.result: engine's name in SHOW ENGINE MARIA LOGS changed. mysql-test/r/maria-recover.result: result for new test. We see corruption messages at first SELECT and then none at second SELECT, expected. mysql-test/r/maria-recovery.result: result update mysql-test/r/maria.result: new variables show up mysql-test/t/disabled.def: BUG#34911 is not fixed but the test had been made independent of the bug (workaround). A new bug (crash) has popped recently, so it has to stay disabled (BUG#35107). mysql-test/t/maria-preload.test: Work around BUG#34911 "FLUSH STATUS doesn't flush what it should": compute differences in status variables before and after relevant queries mysql-test/t/maria-recover-master.opt: test --maria-recover mysql-test/t/maria-recover.test: Test of the --maria-recover option (build a corrupted table and see if it is auto-repaired) mysql-test/t/maria-recovery-big.test: update for new API of include/maria*.inc mysql-test/t/maria-recovery-bitmap.test: update for new API of include/maria*.inc mysql-test/t/maria-recovery.test: update for new API of include/maria*.inc. Corrupted table t1 renamed to t_corrupted1, so that mtr_report.pl does not blindly remove all corruption messages for t1 which is a common name. storage/maria/ha_maria.cc: Enabling maria-recover. Adding option and global variable --maria_force_start_after_recovery_failures: ha_maria_init() calls mark_recovery_start() and mark_recovery_success() to keep track of failed consecutive recoveries and remove logs if needed. Removed \0 in the output of SHOW ENGINE MARIA LOGS; removed hard-coded engine name there. storage/maria/ma_checkpoint.c: new prototype storage/maria/ma_control_file.c: Storing in one byte in the control file, the number of consecutive recovery failures. storage/maria/ma_control_file.h: new prototype storage/maria/ma_init.c: new prototype storage/maria/ma_locking.c: Need to update open_count on disk at first write and close for transactional tables, like we already did for non-transactional tables, otherwise we cannot notice that the table is dubious. storage/maria/ma_loghandler.c: translog_is_log_files() is made more generic to serve either to search or to delete logs (the latter is for --maria-force-start-after-recovery-failures). It also had a bug (always returned FALSE). storage/maria/ma_loghandler.h: export function because ha_maria::mark_recovery_start() needs it storage/maria/ma_recovery.c: changing name of maria_recover() to distinguish from the maria-recover option. storage/maria/ma_recovery.h: changing name of maria_recover() to distinguish from the maria-recover option. storage/maria/ma_test_force_start.pl: Test of --maria-force-start-after-recovery-failures (and also, to be realistic, of --maria-recover). This is standalone because mysql-test-run does not support testing that multiple mysqld restarts expectedly failed. I'll have to run it on my machine and also on a Windows machine. storage/maria/unittest/ma_control_file-t.c: adding recovery_failures to the test storage/maria/unittest/ma_test_loghandler_multigroup-t.c: fix for compiler warning (unused variable in non-debug build) |
||
|
unknown
|
f83bd712ae |
Merge bk-internal.mysql.com:/home/bk/mysql-maria
into mysql.com:/home/my/mysql-maria mysql-test/r/maria.result: Auto merged mysql-test/suite/ndb/r/ndb_auto_increment.result: Auto merged mysql-test/t/maria.test: Auto merged mysys/hash.c: Auto merged mysys/thr_lock.c: Auto merged sql/field.cc: Auto merged sql/ha_ndbcluster.cc: Auto merged sql/ha_ndbcluster.h: Auto merged sql/ha_partition.cc: Auto merged sql/ha_partition.h: Auto merged sql/handler.cc: Auto merged sql/handler.h: Auto merged sql/log_event.cc: Auto merged sql/log_event_old.cc: Auto merged sql/mysqld.cc: Auto merged sql/protocol.cc: Auto merged sql/sql_load.cc: Auto merged sql/sql_parse.cc: Auto merged sql/sql_select.cc: Auto merged sql/sql_table.cc: Auto merged sql/sql_yacc.yy: Auto merged storage/csv/ha_tina.cc: Auto merged storage/federated/ha_federated.cc: Auto merged storage/maria/Makefile.am: Auto merged storage/maria/ma_check.c: Auto merged storage/maria/ma_control_file.c: Auto merged storage/maria/ma_delete_all.c: Auto merged storage/maria/ma_dynrec.c: Auto merged storage/maria/ma_init.c: Auto merged storage/maria/ma_key_recover.c: Auto merged storage/maria/ma_open.c: Auto merged storage/maria/ma_page.c: Auto merged storage/maria/ma_range.c: Auto merged storage/maria/ma_recovery.c: Auto merged storage/maria/ma_test1.c: Auto merged storage/maria/maria_read_log.c: Auto merged storage/maria/unittest/ma_test_all-t: Auto merged storage/maria/unittest/ma_test_loghandler_multigroup-t.c: Auto merged storage/maria/unittest/ma_test_recovery.pl: Auto merged storage/myisam/ha_myisam.cc: Auto merged storage/myisam/myisamdef.h: Auto merged include/my_base.h: Manual merge where error code are kept same as in 5.1 mysys/my_handler.c: No changes sql/item.cc: Manual merge sql/sql_class.cc: Manual merge sql/sql_insert.cc: Manual merge storage/maria/ha_maria.cc: Manual merge storage/maria/ma_blockrec.c: Manual merge storage/maria/ma_delete.c: Manual merge storage/maria/ma_write.c: Manual merge |
||
|
unknown
|
126c1228f5 |
Added versioning of row data
Will in future changeset (soon) av versioning of status variables (number of rows) and index Changed some LEX_STRING to LEX_CUSTRING to avoid casts and warnings Removed some not needed variables (as noticed by Guilhem) include/maria.h: Added prototypes for maria_chk_init_for_check(), maria_versioning() and maria_ignore_trids() include/my_base.h: Add new error HA_ERR_ROW_NOT_VISIBLE include/myisamchk.h: Added variables for checking visibility of rows during maria_chk include/thr_lock.h: Changed argument type from int to my_bool for get_status Added variable allow_multiple_concurrent_insert, to signal if table supports multiple concurrent inserts mysql-test/r/maria-page-checksum.result: Added missing drop table mysql-test/t/maria-page-checksum.test: Added missing drop table mysys/my_handler.c: Added new error messages mysys/thr_lock.c: Added support for multiple concurrent inserts, if table handler supports it sql/sql_yacc.yy: Added LOCK TABLE table_name WRITE CONCURRENT This was added (temporarly?) to be able to check versioning with Maria storage/csv/ha_tina.cc: Updated parameter for get_status storage/maria/ha_maria.cc: Added calls to maria_chk_init_status() Fixed call to ma_control_file_open() storage/maria/ma_blockrec.c: Changed some LEX_STRING to LEX_CUSTRING to avoid casts and warnings Changed back some 'header' parameters to const char* Removed some casts Added support for versioning: - If info->row_flag & ROW_FLAG_TRANSID is set, store transaction id together with the row - When reading rows, check if rows are visible. Give error if not - When scanning table, ignore not visible rows - Added function parameters to some functions, to be able to call _ma_compact_block_page() with different parameters depending of if the page is a HEAD or TAIL page - _ma_compact_block_page() deletes transaction id's that are visible by all running transactions - Added functions for thr_lock() to enable multiple concurrent inserts - Added helper function 'mysql_versioning()' to enable/disable versioning - Added helper function maria_ignore_trids(), used by maria_chk and maria_pack to see all rows. storage/maria/ma_blockrec.h: Updated parameters for some functions. Added new functions to read/store state with thr_lock storage/maria/ma_check.c: Enable handling of transaction id's in rows Give a readable error if a table contains a transation id that makes rows not visible storage/maria/ma_control_file.c: Added option to not give warning if control file doesn't exists. storage/maria/ma_control_file.h: Updated parameter lists for ma_control_file_open() storage/maria/ma_delete.c: Removed not used variable (suggestion by Guilhem) storage/maria/ma_locking.c: Changed type of argument from int -> my_bool storage/maria/ma_open.c: Removed not used variables 'key_write_undo_lsn' and 'key_delete_undo_lsn' Added new thr_lock interface functions for BLOCK_RECORD to enable multiple concurrent insert storage/maria/ma_test1.c: Added option --versioning (-C) to check versioning storage/maria/ma_test2.c: Added option -C to check versioning storage/maria/ma_test_recovery: Forward argumetns to ma_test_recovery.pl storage/maria/ma_write.c: Removed not used variable key_write_undo_lsn storage/maria/maria_chk.c: Always read control file (if exist) at start Initialize checking of tables by calling maria_chk_init_for_check() In verbose mode and in case of error, print max found transaction id storage/maria/maria_def.h: Added Trid to MARIA_ROW to be able to check transaction id for found row Moved 'base_length' from MARIA_ROW to MARIA_HA to be able to handle different base length (with and without TRANSID) without if's Added default row_flag to MARIA_HA for the same reason Changed LEX_STRING -> LEX_CUSTRING to avoid casts in ma_blockrec.c Removed not needed variables key_write_undo_lsn and key_delete_undo_lsn Added prototypes for new functions and fixed those that had changed storage/maria/maria_pack.c: Ensure we can read all rows from the file, independent of the used transaction id storage/maria/maria_read_log.c: Updated arguments to ma_control_file_open() storage/maria/trnman.c: If we have only one transaction, fixed that min_read_from contains current transaction Fixed that trnman_can_read_from() returns that row is readable if it was written by current transaction storage/maria/unittest/ma_control_file-t.c: Updated arguments to ma_control_file_open() storage/maria/unittest/ma_test_all-t: Added test of versioning Removed printing of one extra space storage/maria/unittest/ma_test_loghandler-t.c: Updated arguments to ma_control_file_open() storage/maria/unittest/ma_test_loghandler_first_lsn-t.c: Updated arguments to ma_control_file_open() storage/maria/unittest/ma_test_loghandler_max_lsn-t.c: Updated arguments to ma_control_file_open() storage/maria/unittest/ma_test_loghandler_multigroup-t.c: Updated arguments to ma_control_file_open() storage/maria/unittest/ma_test_loghandler_multithread-t.c: Updated arguments to ma_control_file_open() storage/maria/unittest/ma_test_loghandler_noflush-t.c: Updated arguments to ma_control_file_open() storage/maria/unittest/ma_test_loghandler_nologs-t.c: Updated arguments to ma_control_file_open() storage/maria/unittest/ma_test_loghandler_pagecache-t.c: Updated arguments to ma_control_file_open() storage/maria/unittest/ma_test_loghandler_purge-t.c: Updated arguments to ma_control_file_open() storage/maria/unittest/ma_test_recovery.expected: Updated file with result from new tests storage/maria/unittest/ma_test_recovery.pl: Added options --abort-on-error and --verbose In case of --verbose, print all excuted shell commands Added test of versioning storage/myisam/mi_locking.c: Updated type of parameter storage/myisam/myisamdef.h: Updated type of parameter mysql-test/r/maria-mvcc.result: New BitKeeper file ``mysql-test/r/maria-mvcc.result'' mysql-test/t/maria-mvcc.test: New BitKeeper file ``mysql-test/t/maria-mvcc.test'' |
||
|
unknown
|
b765997467 |
WL#3072 - Maria recovery
When applying a REDO_INDEX_[FREE|NEW]_PAGE, update state members only if record is newer than current state. Print a warning when an old Maria is zero-ing out portions of the control file which it does not know. storage/maria/ma_control_file.c: Print a warning when an old Maria is zero-ing out portions of the control file which it does not know; it may help us when helping troubleshoot customers' and users' problems. storage/maria/ma_key_recover.c: When applying a REDO_INDEX_[FREE|NEW]_PAGE, we should update the state members (key_root, key_del) only if this REDO is newer than the current state. We cannot just update it unconditionally, because that may change it to an old state, which may not be later corrected if all later REDOs are entirely skipped due to their page not being in the dirty pages list of the checkpoint record. |
||
|
unknown
|
2d327efd89 |
Fixing the little bug that if a new version reads an old control file,
it couldn't entirely write its new information (it could not write more bytes than the old format wrote). storage/maria/ma_control_file.c: Let ma_control_file_write_and_force() return error if called when file is not open. When the file was of an old version (no room for our changeable part) we upgrade this part. When the file was of a new version we re-use its format but zero the parts which we can't maintain. storage/maria/ma_init.c: ma_control_file_write_and_force() now does not assert if called when control file is not open. storage/maria/ma_recovery.c: missing ";" (thanks Peter Zaitsev for the bug report and Monty for the patch) |
||
|
unknown
|
722a8ebe5b |
Store maximum transaction id into control file at clean shutdown.
This can serve to maria_chk to check that trids found in rows and keys are not too big. Also used by Recovery when logs are lost. Options --require-control-file, --datadir, --log-dir (yes, the dashes are inconsistent but I imitated mysqld --datadir and --maria-log-dir) for maria_chk. Lock control file _before_ reading its content. storage/maria/ha_maria.cc: new prototype storage/maria/ma_check.c: A function to find the max trid in the system (consults transaction manager and control file), to check tables. storage/maria/ma_checkpoint.c: new prototype storage/maria/ma_control_file.c: Store max trid into control file, in a backward-compatible way (can still read old control files). Parameter to ma_control_file_open(), to not create the log if it's missing (maria_chk needs that). Lock control file _before_ reading its content. Fix for a segfault when reading an old control file (bzero() with a negative second argument) storage/maria/ma_control_file.h: changes to the control file module's API storage/maria/ma_init.c: When Maria shuts down cleanly, store max trid into control file. storage/maria/ma_loghandler.c: new prototype storage/maria/ma_recovery.c: During recovery, consult max trid stored in control file, in case it is bigger than what we found in log (case of logs manually removed by user). storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/maria_chk.c: New option --require-control-file (abort if control file not found), --datadir (path for control file (and for logs if --log-dir not specified)), --log-dir (path for logs). Try to open control file when maria_chk starts. storage/maria/maria_read_log.c: new prototype storage/maria/trnman.c: A new function to know max trid in transaction manager storage/maria/trnman_public.h: New function storage/maria/unittest/ma_control_file-t.c: new prototypes. Testing storing and retrieving the max trid to/from control file storage/maria/unittest/ma_test_loghandler-t.c: new prototype storage/maria/unittest/ma_test_loghandler_first_lsn-t.c: new prototype storage/maria/unittest/ma_test_loghandler_max_lsn-t.c: new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: new prototype storage/maria/unittest/ma_test_loghandler_noflush-t.c: new prototype storage/maria/unittest/ma_test_loghandler_nologs-t.c: new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: new prototype storage/maria/unittest/ma_test_loghandler_purge-t.c: new prototype |
||
|
unknown
|
3057378698 |
Fix for failures of ma_control_file-t, ma_test_loghandler_purge-t,
ma_test_loghandler_pagecache-t on Windows. storage/maria/ma_control_file.c: stat() is unreliable on Windows (does not reflect process' own writes) storage/maria/ma_loghandler.c: translog_set_lsn_for_files() didn't close its file descriptor; it was a real problem as non-closed files could not be purged. Same for translog_truncate_log() in case of error. storage/maria/unittest/ma_test_loghandler_pagecache-t.c: stat() is unreliable on Windows (does not reflect process' own writes) |
||
|
unknown
|
b03e603585 |
Merge oleksandr-byelkins-powerbook-g4-15.local:/Users/bell/mysql/bk/mysql-maria
into oleksandr-byelkins-powerbook-g4-15.local:/Users/bell/mysql/bk/work-maria-wrn storage/maria/ma_blockrec.c: Auto merged storage/maria/ma_check.c: Auto merged storage/maria/ma_control_file.c: Auto merged storage/maria/ma_key_recover.c: Auto merged storage/maria/ma_page.c: Auto merged storage/maria/ma_pagecache.c: Auto merged storage/maria/ma_recovery.c: Auto merged |
||
|
unknown
|
311d8e898f |
Mac compiler warnings fixed.
include/maria.h: mac compiler warnings fixed. storage/maria/ma_bitmap.c: mac compiler warnings fixed. storage/maria/ma_blockrec.c: mac compiler warnings fixed. storage/maria/ma_check.c: mac compiler warnings fixed. storage/maria/ma_control_file.c: mac compiler warnings fixed. storage/maria/ma_create.c: mac compiler warnings fixed. storage/maria/ma_delete.c: mac compiler warnings fixed. storage/maria/ma_ft_boolean_search.c: mac compiler warnings fixed. storage/maria/ma_page.c: mac compiler warnings fixed. storage/maria/ma_pagecache.c: mac compiler warnings fixed. storage/maria/ma_recovery.c: mac compiler warning fixed. storage/maria/ma_rt_test.c: mac compiler warnings fixed. storage/maria/ma_search.c: mac compiler warning fixed. storage/maria/ma_write.c: mac compiler warnings fixed. storage/maria/unittest/ma_control_file-t.c: mac compiler warnings fixed. storage/maria/unittest/ma_pagecache_consist.c: mac compiler warnings fixed. storage/maria/unittest/ma_pagecache_single.c: mac compiler warnings fixed. storage/maria/unittest/ma_test_loghandler_pagecache-t.c: mac compiler warning fixed. storage/maria/unittest/test_file.c: mac compiler warning fixed. |
||
|
unknown
|
3e9e629c59 |
ma_control_file.c:
SCCS merged storage/maria/ma_control_file.c: SCCS merged |
||
|
unknown
|
7300af848a |
Fixes for running maria-recovery*.test and maria-purge.test under
Windows.
include/my_dbug.h:
a DBUG expression to force a flush of the trace file then an abort of the process
mysql-test/include/wait_until_connected_again.inc:
mysqladmin waits for pid file to be gone only under Unix; so
maria_empty_logs.inc cannot wait for mysqld to be gone, so
wait_until_connected_again.inc may send its "show status" to a
not-yet-dead server hence the 1053 error ("server shutdown in progress")
mysys/my_thr_init.c:
overload abort() under Windows, to not have an annoying CRT popup
("ignore/abort/retry" buttons) each time a test intentionally
crashes mysqld
sql/handler.cc:
use new expression
sql/log.cc:
use new expression
sql/mysql_priv.h:
use new expression
storage/maria/ha_maria.cc:
use new expression
storage/maria/ma_blockrec.c:
use new expression
storage/maria/ma_check.c:
use new expression
storage/maria/ma_checkpoint.c:
use new expression
storage/maria/ma_control_file.c:
Can't yet lock control file under Windows (test suite problems,
plus concerns about stray lock preventing a fast restart after crash).
storage/maria/ma_loghandler.c:
A file which should be closed, otherwise translog_purge() (the caller)
cannot delete logs.
|
||
|
unknown
|
7223b4edf5 |
Lock control file also when we have created it in this run. Assertions.
storage/maria/ma_control_file.c: Even if the file is just created we have to lock it, otherwise some other Maria instance may write to it. storage/maria/ma_pagecache.c: catch too big page numbers when they enter the pagecache; this helps debugging BUG#34250. |
||
|
unknown
|
9dd2d08c6e |
Fixed syncronization of buffer flush.
Debug asserts/prints from Monty. No check of mutex operations (by Monty). storage/maria/ma_control_file.c: New external function to check mutex ownership. storage/maria/ma_loghandler.h: No check of mutex operations (by Monty). New external function to check mutex ownership. |
||
|
unknown
|
cc589bef15 |
Fixed bug in undo_key_delete; Caused crashed key files in recovery
Maria is now used for internal temporary tables in MySQL Better usage of VARCHAR and long strings in temporary tables Use packed fields if BLOCK_RECORD is used null_bytes are not anymore stored in a separate field New interface to remember and restore scan position Fixed bugs in unique handling Don't sync Maria temporary tables Lock control file while it's used to stop several processes from using it Changed value of MA_DONT_OVERWRITE_FILE as it collided with MY_SYNC_DIR Split MY_DONT_WAIT into MY_NO_WAIT and MY_SHORT_WAIT (for my_lock()) Added MY_FORCE_LOCK include/my_sys.h: Changed value of MA_DONT_OVERWRITE_FILE as it collided with MY_SYNC_DIR Split MY_DONT_WAIT into MY_NO_WAIT and MY_SHORT_WAIT (for my_lock()) Added MY_FORCE_LOCK include/myisam.h: Make MyISAM columndef compile time compatible with Maria mysql-test/lib/mtr_process.pl: Removed confusing warning (It's common that there is a lot of other files than pid files) mysql-test/mysql-test-run.pl: Added --sync-frm to speed up tests mysql-test/r/maria-recovery.result: Updated results from wrong push mysql-test/suite/rpl/t/rpl_innodb_bug28430.test: Marked test as --big mysys/my_lock.c: If MY_FORCE_LOCK is given, use locking even if my_disable_locking is given If MY_NO_WAIT is given, return at once if lock is occupied If MY_SHORT_WAIT is given, wait some time for lock before returning (This was called MY_DONT_WAIT before) mysys/my_thr_init.c: Fix that we don't give name to thread before it's properly initied sql/handler.cc: Added myisam.h sql/handler.h: Changes to use Maria for internal temporary tables Removed not needed argument to restart_rnd_next() Added function remember_rnd_pos() sql/my_lock.c: If MY_FORCE_LOCK is given, use locking even if my_disable_locking is given If MY_NO_WAIT is given, return at once if lock is occupied If MY_SHORT_WAIT is given, wait some time for lock before returning (This was called MY_DONT_WAIT before) sql/mysql_priv.h: Added maria_hton sql/sql_class.h: Changes to use Maria for internal temporary tables sql/sql_select.cc: Changes to use Maria for internal temporary tables Temporary tables didn't properly switch to dynamic row format if long strings was used Better usage of VARCHAR in temporary tables Use new interface to restart scan in duplicate removal sql/sql_select.h: Changes to use Maria for internal temporary tables sql/sql_show.cc: Changes to use Maria for internal temporary tables Removed all end space sql/sql_table.cc: Set HA_OPTION_PACK_RECORD if we are not using default or static record sql/sql_union.cc: If MY_FORCE_LOCK is given, use locking even if my_disable_locking is given If MY_NO_WAIT is given, return at once if lock is occupied If MY_SHORT_WAIT is given, wait some time for lock before returning (This was called MY_DONT_WAIT before) sql/sql_update.cc: If MY_FORCE_LOCK is given, use locking even if my_disable_locking is given If MY_NO_WAIT is given, return at once if lock is occupied If MY_SHORT_WAIT is given, wait some time for lock before returning (This was called MY_DONT_WAIT before) storage/maria/ha_maria.cc: Use packed fields null_bytes are not anymore stored in a separate field Changes to use Maria for internal temporary tables Give warning if we try to do an ALTER TABLE to a unusable row format storage/maria/ha_maria.h: Allow Maria with block format to restart scanning at given position storage/maria/ma_blockrec.c: Added functions to remember and restore scan position Allocate cur_row.extents so that we don't have to do a malloc on first read Fixed bug when using packed row without packed strings Removed unneeded calls to free_full_pages() Fixed unlikely bug when using old bitmap to read head page and head page had gone away Remember row position when doing undo of delete and update row (needed for undo of key delete) storage/maria/ma_blockrec.h: Added functions to remember and restore scan position storage/maria/ma_close.c: Don't sync temporary tables storage/maria/ma_control_file.c: Lock control file while it's used to stop several processes from using it storage/maria/ma_create.c: Fixed bug when using FIELD_NORMAL that was longer than FULL_PAGE_SIZE Fixed bug that casued fields to not be ordered according to offset Fixed bug in unique creation storage/maria/ma_delete.c: Don't write record reference when deleting key. (Rowid is likely to be different when we undo this) storage/maria/ma_dynrec.c: Fixed core dump when comparing records (happended in unique handling) storage/maria/ma_extra.c: MY_DONT_WAIT -> MY_SHORT_WAIT Removed TODO comment. (Was not relevant as all other instances are guranteed to be closed when we the code is excecuted) Added DBUG_ASSERT() to prove above. storage/maria/ma_key_recover.c: CLR's for UNDO_ROW_DELETE and UNDO_ROW_UPDATE now include rowid for the row. This was needed for undo_key_delete to work, as undo of delete row is likely to put row in a new position. undo_delete_key now doesn't include row position storage/maria/ma_open.c: Added virtual functions for remembering and restoring scan position Fixed wrong key search method when using multi-byte character sets (Bug#32705) Store original column number in index file NOTE: Index files are now incompatible with previous versions! (Ok as we haven't yet made a public Maria release) storage/maria/ma_recovery.c: Set info->cur_row.lastpos when reading CLR's for UNDO_ROW_DELETE or UNDO_ROW_UPDATE storage/maria/ma_scan.c: Added default function to remember and restore scan position storage/maria/maria_def.h: Added virtual functions & variables to remember and restore scan position Added MARIA_MAX_CONTROL_FILE_LOCK_RETRY storage/myisam/ha_myisam.cc: Fixed compiler errors as columdef->type is now an enum, not an integer Added functions to remember and restore scan position storage/myisam/ha_myisam.h: Added functions to remember and restore scan position storage/myisam/mi_check.c: MY_DONT_WAIT -> MY_SHORT_WAIT storage/myisam/mi_extra.c: MY_DONT_WAIT -> MY_SHORT_WAIT storage/myisam/mi_open.c: MY_DONT_WAIT -> MY_SHORT_WAIT storage/myisam/myisamdef.h: MY_DONT_WAIT -> MY_SHORT_WAIT |
||
|
unknown
|
b7ae94d8df |
Fix bug with skipped syncing.
Fix of unit tests. 2 new tests for new features. storage/maria/ma_control_file.c: Getting the definition from the file for unit-test. storage/maria/ma_loghandler.c: Fix bug with skipped syncing. storage/maria/unittest/ma_control_file-t.c: Fix of unit test of control file. 2 new tests for new features of control file. |
||
|
unknown
|
6b3743f0aa |
Fixes for redo/undo logging of key pages
New extendable format for maria_log_control file Fixed some compiler warnings include/maria.h: Added maria_disable_logging() and maria_enable_logging() mysql-test/include/maria_verify_recovery.inc: Updated tests now when key redo/undo works mysql-test/r/maria-recovery.result: Updated tests now when key redo/undo works storage/maria/ma_blockrec.c: Use unified CLR code Added rec_lsn for full pages Moved clr write hook to ma_key_recover.c Changed REDO code to keep pages pinned until undo Mark page_link's as changed storage/maria/ma_blockrec.h: Moved write_hook_for_clr_end() to ma_key_recover.c storage/maria/ma_check.c: Changed key check code to use PAGECACHE_READ_UNKNOWN_PAGE Fixed wrong warning when checking files after maria_pack When unpacking files, we have to use new keypos_to_recpos method When doing repair, we can disregard index key file pages in page cache storage/maria/ma_commit.c: Added simple enable/disable logging functions (Needed for recovery) storage/maria/ma_control_file.c: Make maria control file extendable without having to make it incompatible for older versions storage/maria/ma_control_file.h: New error messages Added CONTROL_FILE_VERSION storage/maria/ma_delete.c: Added redo/undo for key pages change_length -> changed_length to make things similar More comments & more DBUG storage/maria/ma_key_recover.c: Unified CLR method Moved here write_hook_for_clr_end() and common keypage log functions Changed REDO to keep pages pinned until undo Changed UNDO code to change key_root under log mutex storage/maria/ma_key_recover.h: New structures and functions storage/maria/ma_loghandler.c: Include needed files storage/maria/ma_open.c: Change maria_open() to use pread() instead of read() storage/maria/ma_page.c: Fixed bug in key_del handling Clear pages if IDENTICAL_PAGES_AFTER_RECOVERY is defined storage/maria/ma_pagecache.c: Indentation and spelling fixes More DBUG Added helper function: pagecache_block_link_to_buffer() storage/maria/ma_pagecache.h: Added pagecache_block_link_to_buffer() storage/maria/ma_recovery.c: Fixed state.changed Fixed that REDO keeps pages pinned until UNDO Some bug fixes from previous commit Fixes for UNDO/REDO of key pages storage/maria/ma_search.c: Fixed packing and storing of keys to provide more information to caller so that we can do efficent REDO logging of the changes. storage/maria/ma_test1.c: Fixed bug with not initialized variable storage/maria/ma_test2.c: Removed not used code storage/maria/ma_test_all.res: Updated results storage/maria/ma_test_all.sh: Changed one test to test more Removed timing tests as not relevant here storage/maria/ma_test_recovery.expected: Updated test result after redo/undo if key pages works storage/maria/ma_test_recovery: Updated test after redo/undo if key pages works storage/maria/ma_write.c: Moved some general log functions to ma_key_recover.c Fixed some bugs in undo Moved ma_log_split() to _ma_split_page() Small changes in some function arguments to be able to do redo logging storage/maria/maria_chk.c: disable logging while doing repair table storage/maria/maria_def.h: New function prototypes Move some structs and functions to ma_key_recover.c storage/maria/unittest/ma_control_file-t.c: Updated with patch from Sanja NOTE: This is not complete and need to be updated to new control file format storage/maria/unittest/ma_test_loghandler-t.c: Fixed compiler warning |
||
|
unknown
|
21fd2a5a36 |
First part of redo/undo for key pages
Added key_nr to st_maria_keydef for faster keyinfo->keynr conversion
For transactional tables, shift record number in keys up with 1 bit to have place to indicate if transid follows
Checksum for MyISAM now ignores NULL and not used part of VARCHAR
Renamed some variables that caused shadow compiler warnings
Moved extra() call when waiting for tables to not be used to after tables are removed from cache.
Fixed crashing bugs when using Maria TEMPORARY tables with TRUNCATE. Removed 'hack' code in sql directory to go around this bug.
pagecache_unlock_by_ulink() now has extra argument to say if page was changed.
Give error message if we fail to open control file
Mark page cache variables as not flushable
include/maria.h:
Made min page cache larger (needed for pinning key page)
Added key_nr to st_maria_keydef for faster keyinfo->keynr conversion
Added write_comp_flag to move some runtime code to maria_open()
include/my_base.h:
Added new error message to be used when handler initialization failed
include/my_global.h:
Renamed dummy to swap_dummy to avoid conflicts with local 'dummy' variables
include/my_handler.h:
Added const to some parameters
mysys/array.c:
More DBUG
mysys/my_error.c:
Fixed indentation
mysys/my_handler.c:
Added const to some parameters
Added missing error messages
sql/field.h:
Renamed variables to avoid variable shadowing
sql/handler.h:
Renamed parameter to avoid variable name conflict
sql/item.h:
Renamed variables to avoid variable shadowing
sql/log_event_old.h:
Renamed variables to avoid variable shadowing
sql/set_var.h:
Renamed variables to avoid variable shadowing
sql/sql_delete.cc:
Removed maria hack for temporary tables
Fixed indentation
sql/sql_table.cc:
Moved extra() call when waiting for tables to not be used to after tables are removed from cache.
This was needed to ensure we don't do a PREPARE_FOR_DROP or similar call while the table is still in use.
sql/table.cc:
Copy page_checksum from share
Removed Maria hack
storage/maria/Makefile.am:
Added new files
storage/maria/ha_maria.cc:
Renamed records -> record_count and info -> create_info to avoid variable name conflicts
Mark page cache variables as not flushable
storage/maria/ma_blockrec.c:
Moved _ma_unpin_all_pages() to ma_key_recover.c
Moved init of info->pinned_pages to ma_open.c
Moved _ma_finalize_row() to maria_key_recover.h
Renamed some variables to avoid variable name conflicts
Mark page_link.changed for blocks we change directly
Simplify handling of undo link when writing LOGREC_UNDO_ROW_INSERT (old code crashed when having redo for index)
storage/maria/ma_blockrec.h:
Removed extra empty line
storage/maria/ma_checkpoint.c:
Remove not needed trnman.h
storage/maria/ma_close.c:
Free pinned pages (which are now always allocated)
storage/maria/ma_control_file.c:
Give error message if we fail to open control file
storage/maria/ma_delete.c:
Changes for redo logging (first part, logging of underflow not yet done)
- Log undo-key-delete
- Log delete of key
- Updated arguments to _ma_fetch_keypage(), _ma_dispose(), _ma_write_keypage(), _ma_insert()
- Added new arguments to some functions to be able to write redo information
- Mark key pages as changed when we write with PAGECACHE_LOCK_LEFT_WRITELOCKED
Remove one not needed _ma_write_keypage() in d_search() when upper level will do the write anyway
Changed 2 bmove_upp() to bmove() as this made code easer to understand
More function comments
Indentation fixes
storage/maria/ma_ft_update.c:
New arguments to _ma_write_keypage()
storage/maria/ma_loghandler.c:
Fixed some DBUG_PRINT messages
Simplify code
Added new log entrys for key page redo
Renamed some variables to avoid variable name shadowing
storage/maria/ma_loghandler.h:
Moved some defines here
Added define for storing key number on key pages
Added new translog record types
Added enum for type of operations in LOGREC_REDO_INDEX
storage/maria/ma_open.c:
Always allocate info.pinned_pages (we need now also for normal key page usage)
Update keyinfo->key_nr
Added virtual functions to convert record position o number to be stored on key pages
Update keyinfo->write_comp_flag to value of search flag to be used when writing key
storage/maria/ma_page.c:
Added redo for key pages
- Extended _ma_fetch_keypage() with type of lock to put on page and address to used MARIA_PINNED_PAGE
- _ma_fetch_keypage() now pin's pages if needed
- Extended _ma_write_keypage() with type of locks to be used
- ma_dispose() now locks info->s->state.key_del from other threads
- ma_dispose() writes redo log record
- ma_new() locks info->s->state.key_del from other threads if it was used
- ma_new() now pins read page
Other things:
- Removed some not needed arguments from _ma_new() and _ma_dispose)
- Added some new variables to simplify code
- If EXTRA_DEBUG is used, do crc on full page to catch not unitialized bytes
storage/maria/ma_pagecache.h:
Applied patch from Sanja to add extra argument to pagecache_unlock_by_ulink() to mark if page was changed
Added some defines for pagecache priority levels that one can use
storage/maria/ma_range.c:
Added new arguments for call to _ma_fetch_keypage()
storage/maria/ma_recovery.c:
- Added hooks for new translog types:
REDO_INDEX, REDO_INDEX_NEW_PAGE, REDO_INDEX_FREE_PAGE, UNDO_KEY_INSERT, UNDO_KEY_DELETE and
UNDO_KEY_DELETE_WITH_ROOT.
- Moved variable declarations to start of function (portability fixes)
- Removed some not needed initializations
- Set only relevant state changes for each redo/undo entry
storage/maria/lockman.c:
Removed end space
storage/maria/ma_check.c:
Removed end space
storage/maria/ma_create.c:
Removed end space
storage/maria/ma_locking.c:
Removed end space
storage/maria/ma_packrec.c:
Removed end space
storage/maria/ma_pagecache.c:
Removed end space
storage/maria/ma_panic.c:
Removed end space
storage/maria/ma_rt_index.c:
Added new arguments for call to _ma_fetch_keypage(), _ma_write_keypage(), _ma_dispose() and _ma_new()
Fixed indentation
storage/maria/ma_rt_key.c:
Added new arguments for call to _ma_fetch_keypage()
storage/maria/ma_rt_split.c:
Added new arguments for call to _ma_new()
Use new keypage header
Added new arguments for call to _ma_write_keypage()
storage/maria/ma_search.c:
Updated comments & indentation
Added new arguments for call to _ma_fetch_keypage()
Made some variables and arguments const
Added virtual functions for converting row position to number to be stored in key
use MARIA_RECORD_POS of record position instead of my_off_t
Record in MARIA_KEY_PARAM how page was changed one key insert (needed for REDO)
storage/maria/ma_sort.c:
Removed end space
storage/maria/ma_statrec.c:
Updated arguments for call to _ma_rec_pos()
storage/maria/ma_test1.c:
Fixed too small buffer to init_pagecache()
Fixed bug when using insert_count and test_flag
storage/maria/ma_test2.c:
Use more resonable pagecache size
Remove not used code
Reset blob_length to fix wrong output message
storage/maria/ma_test_all.sh:
Fixed wrong test
storage/maria/ma_write.c:
Lots of new code to handle REDO of key pages
No logic changes because of REDO code, mostly adding new arguments and adding new code for logging
Added new arguments for calls to _ma_fetch_keypage(), _ma_write_keypage() and similar functions
Move setting of comp_flag in ma_ck_wrte_btree() from runtime to maria_open()
Zerofill new used pages for:
- To remove possible sensitive data left in buffer
- To get idenitical data on pages after running redo
- Better compression of pages if archived
storage/maria/maria_chk.c:
Added information if table is crash safe
storage/maria/maria_def.h:
New virtual function to convert between record position on key and normal record position
Aded mutex and extra variables to handle locking of share->state.key_del
Moved some structure variables to get things more aligned
Added extra arguments to MARIA_KEY_PARAM to be able to remember what was changed on key page on key insert
Added argument to MARIA_PINNED_PAGE to indicate if page was changed
Updated prototypes for functions
Added some structures for signaling changes in REDO handling
storage/maria/unittest/ma_pagecache_single.c:
Updated arguments for changed function calls
storage/myisam/mi_check.c:
Made calc_check_checksum virtual
storage/myisam/mi_checksum.c:
Update checksums to ignore null columns
storage/myisam/mi_create.c:
Mark if table has null column (to know when we have to use mi_checksum())
storage/myisam/mi_open.c:
Added virtual function for calculating checksum to be able to easily ignore NULL fields
storage/myisam/mi_test2.c:
Fixed bug
storage/myisam/myisamdef.h:
Added virtual function for calculating checksum during check table
Removed ha_key_cmp() as this is in handler.h
storage/maria/ma_key_recover.c:
New BitKeeper file ``storage/maria/ma_key_recover.c''
storage/maria/ma_key_recover.h:
New BitKeeper file ``storage/maria/ma_key_recover.h''
storage/maria/ma_key_redo.c:
New BitKeeper file ``storage/maria/ma_key_redo.c''
|
||
|
unknown
|
13d53bf657 |
Merge some changes from sql directory in 5.1 tree
Changed format for REDO_INSERT_ROWS_BLOBS Fixed several bugs in handling of big blobs Added redo_free_head_or_tail() & redo_insert_row_blobs() Added uuid to control file maria_checks now verifies that not used part of bitmap is 0 REDO_PURGE_BLOCKS -> REDO_FREE_BLOCKS Added REDO_FREE_HEAD_OR_TAIL Fixes problem when trying to read block outside of file during REDO include/my_global.h: STACK_DIRECTION is already set by configure mysql-test/r/maria.result: Updated results mysql-test/t/maria.test: Test shrinking of VARCHAR mysys/my_realloc.c: Fixed indentation mysys/safemalloc.c: Fixed indentation sql/filesort.cc: Removed some casts sql/mysqld.cc: Added missing setting of myisam_stats_method_str sql/uniques.cc: Removed some casts storage/maria/ma_bitmap.c: Added printing of bitmap (for debugging) Renamed _ma_print_bitmap() -> _ma_print_bitmap_changes() Added _ma_set_full_page_bits() Fixed bug in ma_bitmap_find_new_place() (affecting updates) when using big files storage/maria/ma_blockrec.c: Changed format for REDO_INSERT_ROWS_BLOBS Fixed several bugs in handling of big blobs Added code to fix some cases where redo when using blobs didn't produce idenital .MAD files as normal usage REDO_FREE_ROW_BLOCKS doesn't anymore change pages; We only mark things free in bitmap Remove TAIL and filler extents from REDO_FREE_BLOCKS log entry. (Fixed some asserts) REDO_PURGE_BLOCKS -> REDO_FREE_BLOCKS Delete tails in update. (Fixed bug when doing update that shrinks blob/varchar length) Fixed bug when doing insert in block outside of file size. Added redo_free_head_or_tail() & redo_insert_row_blobs() Added pagecache_unlock_by_link() when read fails. Much more comments, DBUG and ASSERT entries storage/maria/ma_blockrec.h: Prototypes of new functions Define of SUB_RANGE_SIZE & BLOCK_FILLER_SIZE storage/maria/ma_check.c: Verify that not used part of bitmap is 0 storage/maria/ma_control_file.c: Added uuid to control file storage/maria/ma_loghandler.c: REDO_PURGE_BLOCKS -> REDO_FREE_BLOCKS Added REDO_FREE_HEAD_OR_TAIL storage/maria/ma_loghandler.h: REDO_PURGE_BLOCKS -> REDO_FREE_BLOCKS Added REDO_FREE_HEAD_OR_TAIL storage/maria/ma_pagecache.c: If we write full block, remove error flag for block. (Fixes problem when trying to read block outside of file) storage/maria/ma_recovery.c: REDO_PURGE_BLOCKS -> REDO_FREE_BLOCKS Added REDO_FREE_HEAD_OR_TAIL storage/maria/ma_test1.c: Allow option after 'b' to be compatible with ma_test2 (This is just to simplify test scripts like ma_test_recovery) storage/maria/ma_test2.c: Default size of blob is now 1000 instead of 1 storage/maria/ma_test_all.sh: Added test for bigger blobs storage/maria/ma_test_recovery.expected: Updated results storage/maria/ma_test_recovery: Added test for bigger blobs |
||
|
unknown
|
e27890cab0 |
WL#3072 Maria recovery
* create page cache before initializing engine and not after, because Maria's recovery needs a page cache * make the creation of a bitmap page more crash-resistent * bugfix (see ma_blockrec.c) * back to old way: create an 8k bitmap page when creating table * preparations for the UNDO phase: recreate TRNs * preparations for Checkpoint: list of dirty pages, testing of rec_lsn to know if page should be skipped during Recovery (unused in this patch as no Checkpoint module pushed yet) * maria_chk tags repaired table with a special LSN * reworking all around in ma_recovery.c (less duplication) mysys/my_realloc.c: noted an issue in my_realloc() sql/mysqld.cc: page cache needs to be created before engines are initialized, because Maria's initialization may do a recovery which needs the page cache. storage/maria/ha_maria.cc: update to new prototype storage/maria/ma_bitmap.c: when creating the first bitmap page we used chsize to 8192 bytes then pwrite (overwrite) the last 2 bytes (8191-8192). If crash between the two operations, this leaves a bitmap page full without its end marker. A later recovery may try to read this page and find it exists and misses a marker and conclude it's corrupted and fail. Changing the chsize to only 8190 bytes: recovery will then find the page is too short and recreate it entirely. storage/maria/ma_blockrec.c: Fix for a bug: when executing a REDO, if the data page is created, data_file_length was increased before _ma_bitmap_set(): _ma_bitmap_set() called _ma_read_bitmap_page() which, due to the increased data_file_length, expected to find a bitmap page on disk with a correct end marker; if the bitmap page didn't exist already in fact, this failed. Fixed by increasing data_file_length only after _ma_read_bitmap_page() has created the new bitmap page correctly. This bug could happen every time a REDO is about creating a new bitmap page. storage/maria/ma_check.c: empty data file has a bitmap page storage/maria/ma_control_file.c: useless parameter to ma_control_file_create_or_open(), just test if this is recovery. storage/maria/ma_control_file.h: new prototype storage/maria/ma_create.c: Back to how it was before: maria_create() creates an 8k bitmap page. Thus (bugfix) data_file_length needs to reflect this instead of being 0. storage/maria/ma_loghandler.c: as ma_test1 and ma_test2 now use real transactions and not dummy_transaction_object, REDO for INSERT/UPDATE/DELETE are always about real transactions, can assert this. A function for Recovery to assign a short id to a table. storage/maria/ma_loghandler.h: new function storage/maria/ma_loghandler_lsn.h: maria_chk tags repaired tables with this LSN storage/maria/ma_open.c: * enforce that DMLs on transactional tables use real transactions and not dummy_transaction_object. * test if table was repaired with maria_chk (which has to been seen as an import of an external table into the server), test validity of create_rename_lsn (header corruption detection) * comments. storage/maria/ma_recovery.c: * preparations for the UNDO phase: recreate TRNs * preparations for Checkpoint: list of dirty pages, testing of rec_lsn to know if page should be skipped during Recovery (unused in this patch as no Checkpoint module pushed yet) * reworking all around (less duplication) storage/maria/ma_recovery.h: a parameter to say if the UNDO phase should be skipped storage/maria/maria_chk.c: tag repaired tables with a special LSN storage/maria/maria_read_log.c: * update to new prototype * no UNDO phase in maria_read_log for now storage/maria/trnman.c: * a function for Recovery to create a transaction (TRN), needed in the UNDO phase * a function for Recovery to grab an existing transaction, needed in the UNDO phase (rollback all existing transactions) storage/maria/trnman_public.h: new functions |
||
|
unknown
|
6bbca54d7d |
WL#3072 - Maria recovery
Unit test for recovery: runs ma_test1 and ma_test2 (both only with INSERTs and DELETEs; UPDATEs disabled as not handled by recovery) then moves the tables elswhere; recreates tables from the log, and compares and fails if there is a difference. Passes now. Most of maria_read_log.c moved to ma_recovery.c, as it will be re-used for recovery-from-ha_maria. Bugfixes of applying of REDO_INSERT, REDO_PURGE_ROW. Applying of REDO_PURGE_BLOCKS, REDO_DELETE_ALL, REDO_DROP_TABLE, UNDO_ROW_INSERT (in REDO phase only, i.e. just doing records++), UNDO_ROW_DELETE, UNDO_ROW_PURGE. Code cleanups. Monty: please look for "QQ". Sanja: please look for "Sanja". Future tasks: recovery of the bitmap (easy), recovery of the state (make it idempotent), more REDOs (Monty to work on REDO_UPDATE?), UNDO phase... Pushing this cset as it looks safe, contains test and bugfixes which will help Monty implement applying of REDO_UPDATE. sql/handler.cc: typo storage/maria/Makefile.am: Adding ma_test_recovery (which ma_test_all invokes, and which can also be run alone). Most of maria_read_log.c moved to ma_recovery.c storage/maria/ha_maria.cc: comments storage/maria/ma_bitmap.c: fixing comments. 2 -> sizeof(maria_bitmap_marker). Bitmap-related part of _ma_initialize_datafile() moves in bitmap module. Now putting the "bm" signature when creating the first bitmap page (it used to happen only at next open, but that caused an annoying difference when testing Recovery if the original run didn't open the table, and it looks more logical like this: it goes to disk only with its signature correct); see the "QQ" comment towards the _ma_initialize_data_file() call in ma_create.c for more). When reading a bitmap page, verify its signature (happens when normally using the table or when CHECKing it; not when REPAIRing it). storage/maria/ma_blockrec.c: * no need to sync the data file if table is not transactional * Comments, code cleanup (log-related data moved to log-related code block, int5store->page_store). * Store the table's short id into LOGREC_UNDO_ROW_PURGE, like we do for other records (though this record will soon be replaced with a CLR). * If "page" is 1 it means the page which extends from byte page*block_size+1 to (page+1)*block_size (byte number 1 being the first byte of the file). The last byte of the file is data_file_length (same convention). A new page needs to be created if the last byte of the page is beyond the last byte of the file, i.e. (page+1)*block_size+1 > data_file_length, so we correct the test (bug found when testing log applying for ma_test1 -M -T --skip-update). * update the page's LSN when removing a row from it during execution of a REDO_PURGE_ROW record (bug found when testing log applying for ma_test1 -M -T --skip-update). * applying of REDO_PURGE_BLOCKs (limited to a one-page range for now). storage/maria/ma_blockrec.h: new functions. maria_bitmap_marker does not need to be exported. storage/maria/ma_close.c: we can always flush the table's state when closing the last instance of the table. And it is needed for maria_read_log (as it does not use maria_lock_database()). storage/maria/ma_control_file.c: when in Recovery, some assertions should not be used. storage/maria/ma_control_file.h: double-inclusion safe storage/maria/ma_create.c: during recovery, don't log records. Comments. Moving the creation of the first bitmap page to ma_bitmap.c storage/maria/ma_delete_table.c: during recovery, don't log records. Log the end-zero of the dropped table's name, so that recovery can use the string in place without extending it to fit an end zero. storage/maria/ma_loghandler.c: * inwrite_rec_hook also needs access to the MARIA_SHARE, like prewrite_rec_hook. This will be needed to update share->records_diff (in the upcoming patch "recovery of the state"). * LOG_DESC::record_ends_group changed to an enum. * LOG_DESC for LOGREC_REDO_PURGE_BLOCKS and LOGREC_UNDO_ROW_PURGE corrected * Sanja please see the @todo LOG BUG * avoiding DBUG_RETURN(func()) as it gives confusing debug traces. storage/maria/ma_loghandler.h: - log write hooks called while the log's lock is held (inwrite_rec_hook) now need the MARIA_SHARE, like prewrite_rec_hook already had - instead of a bool saying if this record's type ends groups or not, we refine: it may not end a group, it may end a group, or it may be a group in itself. Imagine that we had a physical write failure to a table before we log the UNDO, we still end up in external_lock(F_UNLCK) and then we log a COMMIT: we don't want to consider this COMMIT as ending the group of REDOs (don't want to execute those REDOs during Recovery), that's why we say "COMMIT is a group in itself, it aborts any previous group". This also gives one more sanity check in maria_read_log. storage/maria/ma_recovery.c: New Recovery code, replacing the old pseudocode. Most of maria_read_log moved here. Call-able from ha_maria, but not enabled yet. Compared to the previous version of maria_read_log, some bugs have been fixed, debugging output can go to stdout or a disk file (for now it's useful for me, later it can be changed), execution of REDO_DROP_TABLE, REDO_DELETE_ALL, REDO_PURGE_BLOCKS has been added. Duplicate code has been factored into functions. We abort an unfinished group of records if we see a record which is a group in itself (like COMMIT). No need for maria_panic() after a bug (which caused tables to not be closed) was fixed; if there is yet another bug I prefer to see it. When opening a table for Recovery, set data_file_length and key_file_length to their real physical value (these are the easiest state members to restore :). Warn us if the last page was truncated (but Recovery handles it). MARIA_SHARE::state::state::records is now partly recovered (not idempotent, but works if recreating tables from scracth). When applying a REDO to a page, stamp it with the UNDO's LSN (current_group_end_lsn), not with the REDO's LSN; it makes the table more identical to the original table (easier to compare the two tables in the end). Big thing missing: some types of REDOs are not handled, and the UNDO phase does not exist (missing functions to execute UNDOs to actually rollback). So for now tests are only inserting/deleting a few 100 rows, closing the table and seeing if the log is applied ok; it works. UPDATE not handled. storage/maria/ma_recovery.h: new functions: ma_recover() for recovery from inside ha_maria; _ma_apply_log() for maria_read_log (ma_recover() calls _ma_apply_log()). Btw, we need to not use the word "recover" for REPAIR/maria_chk anymore. storage/maria/ma_rename.c: don't write log records during recovery storage/maria/ma_test2.c: - fail if maria_info() or other subtests find some wrong information - new option -g to skip updates. - init the translog before creating the table, so that log applying can work. - in "#if 0" you'll see some fixed bugs (will be removed). storage/maria/ma_test_all.sh: cleanup files. Test log applying. storage/maria/maria_read_log.c: most of the logic moves to ma_recovery.c to be shared between maria_read_log and recovery-from-inside-mysqld. See ma_recovery.c for additional changes made to the moved code. storage/maria/ma_test_recovery: unit test for Recovery. Tests insert and delete, REDO_UPDATE not yet coded. Script is called from ma_test_all. Can run standalone. |
||
|
unknown
|
adac9798bf |
WL#3072 Maria Recovery
- new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that. |
||
|
unknown
|
1a96259191 |
- WL#3239 "log CREATE TABLE in Maria"
- WL#3240 "log DROP TABLE in Maria" - similarly, log RENAME TABLE, REPAIR/OPTIMIZE TABLE, and DELETE no_WHERE_clause (== the DELETE which just truncates the files) - create_rename_lsn added to MARIA_SHARE's state - all these operations (except DROP TABLE) also update the table's create_rename_lsn, which is needed for the correctness of Recovery (see function comment of _ma_repair_write_log_record() in ma_check.c) - write a COMMIT record when transaction commits. - don't log REDOs/UNDOs if this is an internal temporary table like inside ALTER TABLE (I expect this to be a big win). There was already no logging for user-created "CREATE TEMPORARY" tables. - don't fsync files/directories if the table is not transactional - in translog_write_record(), autogenerate a 2-byte-id for the table and log the "id->name" pair (LOGREC_FILE_ID); log LOGREC_LONG_TRANSACTION_ID; automatically store the table's 2-byte-id in any log record. - preparations for Checkpoint: translog_get_horizon(); pausing Checkpoint when some dirty pages are unknown; capturing trn->rec_lsn, trn->first_undo_lsn for Checkpoint and log's low-water-mark computing. - assertions, comments. storage/maria/Makefile.am: more files to build storage/maria/ha_maria.cc: - logging a REPAIR log record if REPAIR/OPTIMIZE was successful. - ha_maria::data_file_type does not have to be set in every info() call, just do it once in open(). - if caller said that transactionality can be disabled (like if caller is ALTER TABLE) i.e. thd->transaction.on==FALSE, then we temporarily disable transactionality of the table in external_lock(); that will ensure that no REDOs/UNDOs are logged for this possibly massive write operation (they are not needed, as if any write fails, the table will be dropped). We re-enable in external_lock(F_UNLCK), which in ALTER TABLE happens before the tmp table replaces the original one (which is good, as thus the final table will have a REDO RENAME and a correct create_rename_lsn). - when we commit we also have to write a log record, so trnman_commit_trn() calls become ma_commit() calls - at end of engine's initialization, we are potentially entering a multi-threaded dangerous world (clients are going to be accepted) and so some assertions of mutex-owning become enforceable, for that we set maria_multi_threaded=TRUE (see ma_control_file.c) storage/maria/ha_maria.h: new member ha_maria::save_transactional (see also ha_maria.cc) storage/maria/ma_blockrec.c: - fixing comments according to discussion with Monty - if a table is transactional but temporarily non-transactional (like in ALTER TABLE), we need to give a sensible LSN to the pages (and, if we give 0, pagecache asserts). - translog_write_record() now takes care of storing the share's 2-byte-id in the log record storage/maria/ma_blockrec.h: fixing comment according to discussion with Monty storage/maria/ma_check.c: When REPAIR/OPTIMIZE modify the data/index file, if this is a transactional table, they must sync it; if they remove files or rename files, they must sync the directory, so that everything is durable. This is just applying to REPAIR/OPTIMIZE the logic already implemented in CREATE/DROP/RENAME a few months ago. Adding a function to write a LOGREC_REPAIR_TABLE at end of REPAIR/OPTIMIZE (called only by ha_maria, not by maria_chk), and to update the table's create_rename_lsn. storage/maria/ma_close.c: fix for a future bug storage/maria/ma_control_file.c: ensuring that if Maria is running in multi-threaded mode, anybody wanting to write to the control file and update last_checkpoint_lsn/last_logno owns the log's lock. storage/maria/ma_control_file.h: see ma_control_file.c storage/maria/ma_create.c: when creating a table: - sync it and its directory only if this is a transactional table and there is a log (no point in syncing in maria_chk) - decouple the two uses of linkname/linkname_ptr (for index file and for data file) into more variables, as we need to know all links until the moment we write the LOGREC_CREATE_TABLE. - set share.data_file_type early so that _ma_initialize_data_file() knows it (Monty's bugfix so that a table always has at least a bitmap page when it is created; so data-file is not 0 bytes anymore). - log a LOGREC_CREATE_TABLE; it contains the bytes which we have just written to the index file's header. Update table's create_rename_lsn. - syncing of kfile had been bugified in a previous merge, correcting - syncing of dfile is now needed as it's not empty anymore - in _ma_initialize_data_file(), use share's block_size and not the global one. This is a gratuitous change, both variables are equal, just that I find it more future-proof to use share-bound variable rather than global one. storage/maria/ma_delete_all.c: log a LOGREC_DELETE_ALL record when doing ma_delete_all_rows(); update create_rename_lsn then. storage/maria/ma_delete_table.c: - logging LOGREC_DROP_TABLE; knowing if this is needed, requires knowing if the table is transactional, which requires opening the table. - we need to sync directories only if the table is transactional storage/maria/ma_extra.c: questions storage/maria/ma_init.c: when maria_end() is called, engine is not multithreaded storage/maria/ma_loghandler.c: - translog_inited has to be visible to ma_create() (see how it is used in ma_create()) - checkpoint record will be a single record, not three - no REDO for TRUNCATE (TRUNCATE calls ma_create() internally so will log a REDO_CREATE) - adding REDO for DELETE no_WHERE_clause (fast DELETE of all rows by truncating the files), REPAIR. - MY_WAIT_IF_FULL to wait&retry if a log write hits a full disk - in translog_write_record(), if MARIA_SHARE does not yet have a 2-byte-id, generate one for it and log LOGREC_FILE_ID; automatically store this short id into log records. - in translog_write_record(), if transaction has not logged its long trid, log LOGREC_LONG_TRANSACTION_ID. - For Checkpoint, we need to know the current end-of-log: adding translog_get_horizon(). - For Control File, adding an assertion that the thread owns the log's lock (control file is protected by this lock) storage/maria/ma_loghandler.h: Changes in log records (see ma_loghandler.c). new prototypes, new functions. storage/maria/ma_loghandler_lsn.h: adding a type LSN_WITH_FLAGS especially for TRN::first_undo_lsn, where the most significant byte is used for flags. storage/maria/ma_open.c: storing the create_rename_lsn in the index file's header (in the state, precisely) and retrieving it from there. storage/maria/ma_pagecache.c: - my set_if_bigger was wrong, correcting it - if the first_in_switch list is not empty, it means that changed_blocks misses some dirty pages, so Checkpoint cannot run and needs to wait. A variable missing_blocks_in_changed_list is added to tell that (should it be named missing_blocks_in_changed_blocks?) - pagecache_collect_changed_blocks_with_lsn() now also tells the minimum rec_lsn (needed for low-water mark computation). storage/maria/ma_pagecache.h: see ma_pagecache.c storage/maria/ma_panic.c: comment storage/maria/ma_range.c: comment storage/maria/ma_rename.c: - logging LOGREC_RENAME_TABLE; knowing if this is needed, requires knowing if the table is transactional, which requires opening the table. - update create_rename_lsn - we need to sync directories only if the table is transactional storage/maria/ma_static.c: comment storage/maria/ma_test_all.sh: - tip for Valgrind-ing ma_test_all - do "export maria_path=somepath" before calling ma_test_all, if you want to run ma_test_all out of storage/maria (useful to have parallel runs, like one normal and one Valgrind, they must not use the same tables so need to run in different directories) storage/maria/maria_def.h: - state now contains, in memory and on disk, the create_rename_lsn - share now contains a 2-byte-id storage/maria/trnman.c: preparations for Checkpoint: capture trn->rec_lsn, trn->first_undo_lsn; minimum first_undo_lsn needed to know log's low-water-mark storage/maria/trnman.h: using most significant byte of first_undo_lsn to hold miscellaneous flags, for now TRANSACTION_LOGGED_LONG_ID. dummy_transaction_object is already declared in ma_static.c. storage/maria/trnman_public.h: dummy_transaction_object was declared in all files including trnman_public.h, while in fact it's a single object. new prototype storage/maria/unittest/ma_test_loghandler-t.c: update for new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: update for new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: update for new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: update for new prototype storage/maria/ma_commit.c: function which wraps: - writing a LOGREC_COMMIT record (==commit on disk) - calling trnman_commit_trn() (=commit in memory) storage/maria/ma_commit.h: new header file .tree-is-private: this file is now needed to keep our tree private (don't push it to public trees). When 5.1 is merged into mysql-maria, we can abandon our maria-specific post-commit trigger; .tree_is_private will take care of keeping commit mails private. Don't push this file to public trees. |
||
|
unknown
|
1a77ebd14c |
Maria: a stronger checksum in the control file, to test integrity.
4 bytes, using my_checksum() (the old checksum was one byte and just a sum of the bytes - that was before I saw we have my_checksum :) storage/maria/ma_control_file.c: stronger checksum (4 bytes instead of 1, and using CRC instead of simple byte sum). storage/maria/unittest/ma_control_file-t.c: Checksum is now 4 bytes (total length of control file is now 23), so LSN and LAST_LOGNO move. |
||
|
unknown
|
46922b5125 |
GPL license update (same change as was done for all files in 5.1).
storage/maria/Makefile.am: GPL license update storage/maria/ft_maria.c: GPL license update storage/maria/ha_maria.cc: GPL license update storage/maria/ha_maria.h: GPL license update storage/maria/lockman.c: GPL license update storage/maria/lockman.h: GPL license update storage/maria/ma_bitmap.c: GPL license update storage/maria/ma_blockrec.c: GPL license update storage/maria/ma_blockrec.h: GPL license update storage/maria/ma_cache.c: GPL license update storage/maria/ma_changed.c: GPL license update storage/maria/ma_check.c: GPL license update storage/maria/ma_checkpoint.c: GPL license update storage/maria/ma_checkpoint.h: GPL license update storage/maria/ma_checksum.c: GPL license update storage/maria/ma_close.c: GPL license update storage/maria/ma_control_file.c: GPL license update storage/maria/ma_control_file.h: GPL license update storage/maria/ma_create.c: GPL license update storage/maria/ma_dbug.c: GPL license update storage/maria/ma_delete.c: GPL license update storage/maria/ma_delete_all.c: GPL license update storage/maria/ma_delete_table.c: GPL license update storage/maria/ma_dynrec.c: GPL license update storage/maria/ma_extra.c: GPL license update storage/maria/ma_ft_boolean_search.c: GPL license update storage/maria/ma_ft_eval.c: GPL license update storage/maria/ma_ft_eval.h: GPL license update storage/maria/ma_ft_nlq_search.c: GPL license update storage/maria/ma_ft_parser.c: GPL license update storage/maria/ma_ft_stem.c: GPL license update storage/maria/ma_ft_test1.c: GPL license update storage/maria/ma_ft_test1.h: GPL license update storage/maria/ma_ft_update.c: GPL license update storage/maria/ma_ftdefs.h: GPL license update storage/maria/ma_fulltext.h: GPL license update storage/maria/ma_info.c: GPL license update storage/maria/ma_init.c: GPL license update storage/maria/ma_key.c: GPL license update storage/maria/ma_keycache.c: GPL license update storage/maria/ma_least_recently_dirtied.c: GPL license update storage/maria/ma_least_recently_dirtied.h: GPL license update storage/maria/ma_locking.c: GPL license update storage/maria/ma_open.c: GPL license update storage/maria/ma_packrec.c: GPL license update storage/maria/ma_page.c: GPL license update storage/maria/ma_panic.c: GPL license update storage/maria/ma_preload.c: GPL license update storage/maria/ma_range.c: GPL license update storage/maria/ma_recovery.c: GPL license update storage/maria/ma_recovery.h: GPL license update storage/maria/ma_rename.c: GPL license update storage/maria/ma_rfirst.c: GPL license update storage/maria/ma_rkey.c: GPL license update storage/maria/ma_rlast.c: GPL license update storage/maria/ma_rnext.c: GPL license update storage/maria/ma_rnext_same.c: GPL license update storage/maria/ma_rprev.c: GPL license update storage/maria/ma_rrnd.c: GPL license update storage/maria/ma_rsame.c: GPL license update storage/maria/ma_rsamepos.c: GPL license update storage/maria/ma_rt_index.c: GPL license update storage/maria/ma_rt_index.h: GPL license update storage/maria/ma_rt_key.c: GPL license update storage/maria/ma_rt_key.h: GPL license update storage/maria/ma_rt_mbr.c: GPL license update storage/maria/ma_rt_mbr.h: GPL license update storage/maria/ma_rt_split.c: GPL license update storage/maria/ma_rt_test.c: GPL license update storage/maria/ma_scan.c: GPL license update storage/maria/ma_search.c: GPL license update storage/maria/ma_sort.c: GPL license update storage/maria/ma_sp_defs.h: GPL license update storage/maria/ma_sp_key.c: GPL license update storage/maria/ma_sp_test.c: GPL license update storage/maria/ma_static.c: GPL license update storage/maria/ma_statrec.c: GPL license update storage/maria/ma_test1.c: GPL license update storage/maria/ma_test2.c: GPL license update storage/maria/ma_test3.c: GPL license update storage/maria/ma_unique.c: GPL license update storage/maria/ma_update.c: GPL license update storage/maria/ma_write.c: GPL license update storage/maria/maria_chk.c: GPL license update storage/maria/maria_def.h: GPL license update storage/maria/maria_ftdump.c: GPL license update storage/maria/maria_pack.c: GPL license update storage/maria/tablockman.c: GPL license update storage/maria/tablockman.h: GPL license update storage/maria/trnman.c: GPL license update storage/maria/trnman.h: GPL license update |
||
|
unknown
|
3bc8f629dd |
Postreview changes.
Fixed befaviour when loghandler flags changed from one run to another one. Description of maria transaction log and control file added to the file commandÍs magic number file. mysys/mf_pagecache.c: postreview changes storage/maria/ma_control_file.c: Postreview changes. storage/maria/ma_control_file.h: Postreview changes. storage/maria/ma_loghandler.c: Postreview changes. Fixed befaviour when loghandler flags changed from one run to another one. storage/maria/ma_loghandler.h: Postreview changes. Functions comment left only near the function body. storage/maria/ma_loghandler_lsn.h: Postreview changes. storage/maria/unittest/ma_test_loghandler-t.c: Postreview changes. storage/maria/unittest/ma_test_loghandler_multigroup-t.c: Postreview changes. storage/maria/unittest/ma_test_loghandler_multithread-t.c: Postreview changes. storage/maria/unittest/ma_test_loghandler_pagecache-t.c: Postreview changes. support-files/magic: Description of maria transaction log and control file added to the file commandÕs magic number file. |
||
|
unknown
|
91a8199773 |
Postmerge fix (including changing type of LSN)
Some debug info and comments added include/pagecache.h: postmerge fix mysys/mf_pagecache.c: Postmerge fix (including changing type of LSN) Additional DBUG_ASSERTs added Comment about pinning mechanism added storage/maria/ma_control_file.c: Used the same LSN storing procedure everywhere Postmerge fix (including changing type of LSN) storage/maria/ma_control_file.h: Postmerge fix (including changing type of LSN) storage/maria/ma_loghandler.c: Postmerge fix (including changing type of LSN) storage/maria/ma_loghandler.h: Postmerge fix (including changing type of LSN) storage/maria/ma_loghandler_lsn.h: Postmerge fix (including changing type of LSN) storage/maria/unittest/Makefile.am: Postmerge fix storage/maria/unittest/ma_control_file-t.c: Postmerge fix (including changing type of LSN) storage/maria/unittest/ma_test_loghandler-t.c: Postmerge fix (including changing type of LSN) storage/maria/unittest/ma_test_loghandler_multigroup-t.c: Postmerge fix (including changing type of LSN) storage/maria/unittest/ma_test_loghandler_multithread-t.c: Postmerge fix (including changing type of LSN) storage/maria/unittest/ma_test_loghandler_pagecache-t.c: Postmerge fix (including changing type of LSN) storage/maria/unittest/mf_pagecache_consist.c: Postmerge fix (including changing type of LSN) storage/maria/unittest/mf_pagecache_single.c: Postmerge fix (including changing type of LSN) |
||
|
unknown
|
a41ac15b96 |
Maria - various fixes around durability of files:
1) on Mac OS X >=10.3, fcntl() is recommended over fsync (from the man page: "[With fsync()] the disk drive may also re-order the data so that later writes may be present while earlier writes are not. Applications such as databases that require a strict ordering of writes should use F_FULLFSYNC to ensure their data is written in the order they expect"). I have seen two other pieces of software changing from fsync to F_FULLFSYNC on Mac OS X. 2) to make a file creation/deletion/renaming durable on Linux (at least ext2 as I have tested) (see "man fsync"), a fsync() on the directory is needed: new functions to do that, and a flag MY_SYNC_DIR to do it in my_create/my_delete/my_rename. 3) now using this directory syncing when creating he frm if opt_sync_frm, and for Maria's control file when it is created. include/my_sys.h: new flag to my_create/my_delete/my_rename, which asks to sync the directory after the operation is done (currently does nothing except on Linux) libmysql/CMakeLists.txt: my_create() now depends on my_sync() so my_sync is needed for libmysql libmysql/Makefile.shared: my_create() now depends on my_sync() so my_sync is needed for libmysql mysys/my_create.c: my_create() can now sync the directory if asked for mysys/my_delete.c: my_delete() can now sync the directory if asked for mysys/my_open.c: it was a bug that my_close() is done on fd but a positive fd would still be returned, by my_register_filename(). mysys/my_rename.c: my_rename() can now sync the two directories (the one of "from" and the one of "to") if asked for. mysys/my_sync.c: On recent Mac OS X, fcntl(F_FULLFSYNC) is recommended over fsync() (see "man fsync" on Mac OS X 10.3). my_sync_dir(): to sync a directory after a file creation/deletion/ renaming; can be called directly or via MY_SYNC_DIR in my_create/ my_delete/my_rename(). No-op except on Linux (see "man fsync" on Linux). my_sync_dir_from_file(): same as above, just more practical when the caller has a file name but no directory name ready. Should the #warning even be a #error? I mean do we want to release binaries which don't guarantee any durability? sql/log.cc: a TODO for the future. sql/unireg.cc: If we sync the frm it makes sense to also sync its creation in the directory. storage/maria/ma_control_file.c: control file is vital, try to make it to disk |
||
|
unknown
|
09a7f30973 |
WL#3234 Maria Control file manager
last round of fixes to the storage engines' and plugins' unit tests structure. Will extract a total patch and push it in 5.1 as has been approved. Makefile.am: unittest must be before storage and plugin, because engine and plugin may have unit tests which link with libtap which is found in unitttest. config/ac-macros/plugins.m4: When enabling an engine/plugin, add its directory to the list of directories where unit tests should be searched. That is, its directory will be recursively searched by our unit test framework which will execute any executable *-t file. storage/maria/ma_control_file.c: those my_message pollute the output of unit tests. storage/maria/plug.in: When Maria is enabled, add its unittest Makefile. unittest/Makefile.am: plugins too |
||
|
unknown
|
cdf831cf94 |
WL#3071 Maria checkpoint:
changing pseudocode to use the structures of the Maria pagecache
("pagecache->changed_blocks" etc) and other Maria structures
inherited from MyISAM (THR_LOCK_maria etc).
mysys/mf_pagecache.c:
comment
storage/maria/ma_checkpoint.c:
changing pseudocode to use the structures of the Maria pagecache
("pagecache->changed_blocks" etc) and other Maria structures
inherited from MyISAM (THR_LOCK_maria etc).
storage/maria/ma_checkpoint.h:
copyright
storage/maria/ma_control_file.c:
copyright
storage/maria/ma_control_file.h:
copyright
storage/maria/ma_least_recently_dirtied.c:
copyright
storage/maria/ma_least_recently_dirtied.h:
copyright
storage/maria/ma_recovery.c:
copyright
storage/maria/ma_recovery.h:
copyright
storage/maria/unittest/Makefile.am:
copyright
|
||
|
unknown
|
26fb36067a |
WL#3234 Maria control file manager.
Fitting ma_control_file_test into the mytap unittest framework: new directories: - unittest/storage/ for unit tests of any storage engine - unittest/storage/maria for ... Maria, containing ma_control_file-t. Later, older tests like ma_test*, ma_test_all (but which is Unix dependent in its current form) could move here too. The plugins macro enable building of unittest/storage/X for any enabled engine X which has such a directory. If Falcon wants to have unit tests there too, I may have to merge this patch into 5.x one day. config/ac-macros/plugins.m4: If a storage engine has a directory in unittest/storage, build this directory. configure.in: build storage engines' unit tests. storage/maria/Makefile.am: ma_control_file_test moves to unittest/storage/maria storage/maria/ma_control_file.c: more error codes when opening the control file fails. ma_control_file_end() may now return an error if my_close() failed. storage/maria/ma_control_file.h: more error codes when opening the control file fails. unittest/Makefile.am: adding unit tests for storage engines. Note that unit.pl simply recurses into "storage", so if a unit test for storage engine X has been built previously, and now you re-configure (without making clean) to disable this engine, then the unit test of X will not be rebuilt but will still be present in storage/X, so will be run. unittest/storage/maria/ma_control_file-t.c: Making the test fit the mytap framework (return all the way up the stack instead of assert(); use the mytap functions plan(), ok() etc). Adding test of file too short/long. unittest/storage/maria/Makefile.am: a_control_file-t is added to the Maria unit tests. Later, older tests (ma_test1 etc) could also move here. unittest/storage/Makefile.am: New BitKeeper file ``unittest/storage/Makefile.am'' |
||
|
unknown
|
fa793bca31 |
WL#3234 "Maria - control file manager":
added checksum of the file. Now we have size + magic string + checksum to detect that all is ok. Plus misc fixes for "make dist" to work and the resulting tarball to build include/Makefile.am: adding pagecache.h to help the tarball build. The model of pagecache.h, keycache.h, is in pkginclude_HEADERS, wonder why. Adding pagecache.h to noinst_HEADERS for now. storage/maria/Makefile.am: adding ma_control_file.h to help the tarball build storage/maria/ma_control_file.c: adding a simple checksum to the control file. We protect against corruption of this file like this: - test size - test magic string at start - test checksum I also add some simple my_message() errors (to be changed to a better reporting later). storage/maria/ma_control_file.h: comments storage/maria/ma_control_file_test.c: test of wrong checksum in control file storage/maria/CMakeLists.txt: just to make "make dist" happy for now. |
||
|
unknown
|
a1f25544d5 |
WL#3234 "Maria - control file manager"
- fixes to the control file module - unit test for it - renames of all Maria files I created to start with ma_ storage/maria/ma_checkpoint.c: Rename: storage/maria/checkpoint.c -> storage/maria/ma_checkpoint.c storage/maria/ma_checkpoint.h: Rename: storage/maria/checkpoint.h -> storage/maria/ma_checkpoint.h storage/maria/ma_least_recently_dirtied.c: Rename: storage/maria/least_recently_dirtied.c -> storage/maria/ma_least_recently_dirtied.c storage/maria/ma_least_recently_dirtied.h: Rename: storage/maria/least_recently_dirtied.h -> storage/maria/ma_least_recently_dirtied.h storage/maria/ma_recovery.c: Rename: storage/maria/recovery.c -> storage/maria/ma_recovery.c storage/maria/ma_recovery.h: Rename: storage/maria/recovery.h -> storage/maria/ma_recovery.h storage/maria/Makefile.am: control file module and its unit test program storage/maria/ma_control_file.c: DBUG_ tags. Fix for gcc warnings. log_no -> logno (I felt "_no" sounded like a standalone "No" word). ma_ prefix for some functions. last_checkpoint_lsn_at_startup -> last_checkpoint_lsn (no need to make special vars for the values at startup). Same for last_logno. ma_control_file_write_and_force() now updates last_checkpoint_lsn and last_logno, the idea being that they belong to the module, others should not update them. And thus when the module shuts down, it zeroes those vars. storage/maria/ma_control_file.h: importing structs from Sanja to get the control file module to compile; we'll remove that when Sanja pushes the log handler. CONTROL_FILE_IMPOSSIBLE_LOGNO is 0, not FFFFFFFF. storage/maria/ma_control_file_test.c: Unit test program for the Maria control file module. Modelled after other ma_test* files in this directory (so, does not follow the unit test framework recently introduced with libtap; TODO as a task on all ma_test* programs). We test that writing to the control file works, and re-reading from it too, we check (by reading the file by ourselves) that its content on disk is correct, and check that a corrupted control file is detected. |
Renamed from storage/maria/control_file.c (Browse further)