There were so many changes into mtr (this is the new mtr coming) that I rather
copied mtr from 6.0-main here (at least this one knows how to run Maria tests).
I also fixed suite/maria tests to be accepted by the new mtr.
mysys/thr_mutex.c:
adding DBUG_PRINT here, so that we can locate where the warning is issued.
http://forge.mysql.com/worklog/task.php?id=4374
new option --maria-force-start-after-recovery-failures=N; number of consecutive recovery failures (failures
of log reading or recovery processing, anything in [translog_init(),maria_recovery_from_log()])
is stored in the control file; if at a Maria start they are more than N, logs are removed. This is for automated
systems which have to run whatever happens. As tables risk staying corrupted, --maria-recover should also
be used on them: this revision makes maria-recover work (it was disabled).
Fixed bug in translog_is_log_files(). translog_init() now prints message to error log if failed.
Removed \0 in the output of SHOW ENGINE MARIA LOGS; removed hard-coded engine name there.
KNOWN_BUGS.txt:
As option --maria-force-start-after-recovery-failures is added, it corresponds to the wish "we should fix that if this happens etc".
LOAD INDEX is not ignored since a few weeks. Listed concurrency bugs have been fixed some time ago.
Recovery of fulltext and GIS indexes works since a few weeks.
mysql-test/include/maria_make_snapshot.inc:
configurable prefix in table's name (so far 't' or 't_corrupted')
mysql-test/include/maria_make_snapshot_for_comparison.inc:
configurable prefix in table's name (so far 't' or 't_corrupted')
mysql-test/include/maria_make_snapshot_for_feeding_recovery.inc:
configurable prefix in table's name (so far 't' or 't_corrupted')
mysql-test/include/maria_verify_recovery.inc:
configurable prefix in table's name (so far 't' or 't_corrupted')
mysql-test/lib/mtr_report.pl:
new test maria-recover.test generates expected corruption warnings in the error log. maria-recovery.test's corrupted table is renamed to t_corrupted1 instead of t1.
mysql-test/r/maria-preload.result:
result update. maria_pagecache_read* values are similar to the previous version of this file, though a bit bigger
because using the information_schema and the join leads to some internal maria temp table being used, and thus some
blocks of it being read.
mysql-test/r/maria-purge.result:
engine's name in SHOW ENGINE MARIA LOGS changed.
mysql-test/r/maria-recover.result:
result for new test. We see corruption messages at first SELECT and then none at second SELECT, expected.
mysql-test/r/maria-recovery.result:
result update
mysql-test/r/maria.result:
new variables show up
mysql-test/t/disabled.def:
BUG#34911 is not fixed but the test had been made independent of the bug (workaround). A new bug (crash) has popped recently, so it has to stay
disabled (BUG#35107).
mysql-test/t/maria-preload.test:
Work around BUG#34911 "FLUSH STATUS doesn't flush what it should":
compute differences in status variables before and after relevant queries
mysql-test/t/maria-recover-master.opt:
test --maria-recover
mysql-test/t/maria-recover.test:
Test of the --maria-recover option (build a corrupted table and see if it is auto-repaired)
mysql-test/t/maria-recovery-big.test:
update for new API of include/maria*.inc
mysql-test/t/maria-recovery-bitmap.test:
update for new API of include/maria*.inc
mysql-test/t/maria-recovery.test:
update for new API of include/maria*.inc. Corrupted table t1 renamed to t_corrupted1, so that mtr_report.pl
does not blindly remove all corruption messages for t1 which is
a common name.
storage/maria/ha_maria.cc:
Enabling maria-recover.
Adding option and global variable --maria_force_start_after_recovery_failures: ha_maria_init()
calls mark_recovery_start() and mark_recovery_success() to keep track of failed consecutive recoveries
and remove logs if needed.
Removed \0 in the output of SHOW ENGINE MARIA LOGS; removed hard-coded engine name there.
storage/maria/ma_checkpoint.c:
new prototype
storage/maria/ma_control_file.c:
Storing in one byte in the control file, the number of consecutive recovery failures.
storage/maria/ma_control_file.h:
new prototype
storage/maria/ma_init.c:
new prototype
storage/maria/ma_locking.c:
Need to update open_count on disk at first write and close for transactional tables, like we already did for
non-transactional tables, otherwise we cannot notice that the table is dubious.
storage/maria/ma_loghandler.c:
translog_is_log_files() is made more generic to serve either to search or to delete logs (the latter is
for --maria-force-start-after-recovery-failures). It also had a bug (always returned FALSE).
storage/maria/ma_loghandler.h:
export function because ha_maria::mark_recovery_start() needs it
storage/maria/ma_recovery.c:
changing name of maria_recover() to distinguish from the maria-recover option.
storage/maria/ma_recovery.h:
changing name of maria_recover() to distinguish from the maria-recover option.
storage/maria/ma_test_force_start.pl:
Test of --maria-force-start-after-recovery-failures (and also, to be realistic, of --maria-recover).
This is standalone because mysql-test-run does not support testing that multiple mysqld restarts expectedly failed.
I'll have to run it on my machine and also on a Windows machine.
storage/maria/unittest/ma_control_file-t.c:
adding recovery_failures to the test
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
fix for compiler warning (unused variable in non-debug build)
Fixed typo in maria_read_log "help" output.
Compilation warning on Mac OS fixed.
mysql-test/include/maria_empty_logs.inc:
Added support of moving maria log relatively
to master data directory.
mysql-test/include/maria_make_snapshot_for_feeding_recovery.inc:
Added support of moving maria log relatively
to master data directory.
mysql-test/include/maria_verify_recovery.inc:
Added support of moving maria log relatively
to master data directory.
mysql-test/t/maria-recovery-master.opt:
Check of recovery with logs in other place.
mysql-test/t/maria-recovery.test:
Check of recovery with logs in other place.
storage/maria/ha_maria.cc:
Support of moving logs in other directory then data root.
storage/maria/ma_loghandler.c:
Compilation warning on Mac OS fixed.
storage/maria/ma_static.c:
Support of moving logs in other directory then data root.
storage/maria/ma_test1.c:
Support of moving logs in other directory then data root.
storage/maria/maria_def.h:
Support of moving logs in other directory then data root.
storage/maria/maria_read_log.c:
Support of moving logs in other directory then data root.
Fixed typo in maria_read_log "help" output.
fix for bug: if a crash happened right after writing a REDO like this:
REDO - UNDO - REDO*, then recovery would ignore the last REDO* (ok),
rollback: REDO - UNDO - REDO* - REDO - CLR, and a next recovery would
thus execute REDO* instead of skipping it again. Recovery now logs
LOGREC_INCOMPLETE_GROUP when it meets REDO* for the first time,
to draw a boundary and ensure it is always skipped. Tested by hand.
Note: ma_test_all fails "maria_chk: error: Key 1 - Found too many records"
not due to this patch (failed before).
BitKeeper/triggers/post-commit:
no truncation of the commit mail, or how to review patches?
mysql-test/include/maria_verify_recovery.inc:
let caller choose the statement used to crash (sometimes we
want the crash to happen at special places)
mysql-test/t/maria-recovery.test:
user of maria_verify_recovery.inc now specifies statement which the
script should use for crashing.
storage/maria/ma_bitmap.c:
it's easier to search for all places using functions from the bitmap
module (like in ma_blockrec.c) if those exported functions all start
with "_ma_bitmap": renaming some of them.
Assertion that when we read a bitmap page, overwriting bitmap->map,
we are not losing information (i.e. bitmap->changed is false).
storage/maria/ma_blockrec.c:
update to new names. Adding code (disabled, protected by a #ifdef)
that I use to test certain crash scenarios (more to come).
storage/maria/ma_blockrec.h:
update to new names
storage/maria/ma_checkpoint.c:
update to new names
storage/maria/ma_extra.c:
update to new names
storage/maria/ma_loghandler.c:
new LOGREC_INCOMPLETE_GROUP
storage/maria/ma_loghandler.h:
new LOGREC_INCOMPLETE_GROUP
storage/maria/ma_recovery.c:
When at the end of the REDO phase we have identified some transactions
with incomplete REDO groups (REDOs without an UNDO or CLR_END),
for each of them we log LOGREC_INCOMPLETE_GROUP. This way, the
upcoming UNDO phase can write more records for such transaction,
a future recovery won't pair the incomplete group with the
CLR_END (as there is LOGREC_INCOMPLETE_GROUP to draw a boundary).
New extendable format for maria_log_control file
Fixed some compiler warnings
include/maria.h:
Added maria_disable_logging() and maria_enable_logging()
mysql-test/include/maria_verify_recovery.inc:
Updated tests now when key redo/undo works
mysql-test/r/maria-recovery.result:
Updated tests now when key redo/undo works
storage/maria/ma_blockrec.c:
Use unified CLR code
Added rec_lsn for full pages
Moved clr write hook to ma_key_recover.c
Changed REDO code to keep pages pinned until undo
Mark page_link's as changed
storage/maria/ma_blockrec.h:
Moved write_hook_for_clr_end() to ma_key_recover.c
storage/maria/ma_check.c:
Changed key check code to use PAGECACHE_READ_UNKNOWN_PAGE
Fixed wrong warning when checking files after maria_pack
When unpacking files, we have to use new keypos_to_recpos method
When doing repair, we can disregard index key file pages in page cache
storage/maria/ma_commit.c:
Added simple enable/disable logging functions
(Needed for recovery)
storage/maria/ma_control_file.c:
Make maria control file extendable without having to make it incompatible for older versions
storage/maria/ma_control_file.h:
New error messages
Added CONTROL_FILE_VERSION
storage/maria/ma_delete.c:
Added redo/undo for key pages
change_length -> changed_length to make things similar
More comments & more DBUG
storage/maria/ma_key_recover.c:
Unified CLR method
Moved here write_hook_for_clr_end() and common keypage log functions
Changed REDO to keep pages pinned until undo
Changed UNDO code to change key_root under log mutex
storage/maria/ma_key_recover.h:
New structures and functions
storage/maria/ma_loghandler.c:
Include needed files
storage/maria/ma_open.c:
Change maria_open() to use pread() instead of read()
storage/maria/ma_page.c:
Fixed bug in key_del handling
Clear pages if IDENTICAL_PAGES_AFTER_RECOVERY is defined
storage/maria/ma_pagecache.c:
Indentation and spelling fixes
More DBUG
Added helper function: pagecache_block_link_to_buffer()
storage/maria/ma_pagecache.h:
Added pagecache_block_link_to_buffer()
storage/maria/ma_recovery.c:
Fixed state.changed
Fixed that REDO keeps pages pinned until UNDO
Some bug fixes from previous commit
Fixes for UNDO/REDO of key pages
storage/maria/ma_search.c:
Fixed packing and storing of keys to provide more information to caller so
that we can do efficent REDO logging of the changes.
storage/maria/ma_test1.c:
Fixed bug with not initialized variable
storage/maria/ma_test2.c:
Removed not used code
storage/maria/ma_test_all.res:
Updated results
storage/maria/ma_test_all.sh:
Changed one test to test more
Removed timing tests as not relevant here
storage/maria/ma_test_recovery.expected:
Updated test result after redo/undo if key pages works
storage/maria/ma_test_recovery:
Updated test after redo/undo if key pages works
storage/maria/ma_write.c:
Moved some general log functions to ma_key_recover.c
Fixed some bugs in undo
Moved ma_log_split() to _ma_split_page()
Small changes in some function arguments to be able to do redo logging
storage/maria/maria_chk.c:
disable logging while doing repair table
storage/maria/maria_def.h:
New function prototypes
Move some structs and functions to ma_key_recover.c
storage/maria/unittest/ma_control_file-t.c:
Updated with patch from Sanja
NOTE: This is not complete and need to be updated to new control file format
storage/maria/unittest/ma_test_loghandler-t.c:
Fixed compiler warning
possibility, if requested in the test, to do byte-for-byte comparison
of data files after recovery. This makes sense only when testing
REDOs, as UNDO phase generates new log records thus data pages get
new LSNs. First test of maria-recovery.test is changed to test only
REDOs and do byte-for-byte comparison.
mysql-test/include/maria_make_snapshot.inc:
possibility to compare physically the data files (makes sense
only if testing REDO phase).
mysql-test/include/maria_verify_recovery.inc:
allow physical comparison of data files after recovery if requested
mysql-test/r/maria-recovery.result:
result update (first test tests REDO only now)
mysql-test/t/maria-recovery.test:
first test tests REDO only now, and thus can do byte-for-byte
comparison of data files.
See test maria-recovery.test for a model; all include scripts have
an "API" section at start if they do take parameters from outside.
* Fixing bug reported by Jani and Monty (when two REDOs about the same
page in one group, see ma_blockrec.c).
* Fixing small bugs in recovery
mysql-test/include/wait_until_connected_again.inc:
be sure to enter the loop (the previous query by the caller may not have
failed: it could be
query;
mysqladmin shutdown;
call this script).
mysql-test/lib/mtr_process.pl:
* Through the "expect" file a test can tell mtr that a server crash
is expected. What the file contains is irrelevant. Now if its last
line starts with "wait", mtr will wait before restarting (it will
wait for the last line to not start with "wait"). This is for
tests which need to mangle files under the feet of a dead mysqld.
* Remove "expect" file before restarting; otherwise there could be a
race condition: tests sees server restarted, does something, writes
an "expect" file, and then mtr removes that file, then
test kills mysqld, and then mtr will never restart it.
storage/maria/ma_blockrec.c:
- when applying a REDO in recovery, we don't anymore put UNDO's LSN on the page
at once; indeed if in this REDO's group there comes another REDO
for the same page it would be wrongly skipped. Instead, we keep
pages pinned, don't change their LSN. When done with all REDOs
of the group we unpin them and stamp them with UNDO's LSN.
- fixing bug in applying of REDO_PURGE_BLOCKS in recovery: page_range
sometimes has TAIL_BIT set, need to turn it down to know the real page
range.
- Both bugs are covered in maria-recovery.test
storage/maria/ma_checkpoint.c:
Capability to, in debug builds only, do some special operations
(flush all bitmap and data pages, flush state, flush log)
and crash mysqld, to later test recovery.
Driven by some --debug=d, symbols.
storage/maria/ma_open.c:
debugging info
storage/maria/ma_pagecache.c:
Now that we can _ma_unpin_all_pages() during the REDO phase
to set page's LSN, the assertion needs to be relaxed.
storage/maria/ma_recovery.c:
- open trace file in append mode (useful when a test triggers several
recoveries, we see them all).
- fixing wrong error detection, it's possible that during recovery
we want to open an already open table.
- when applying a REDO in recovery, we don't anymore put UNDO's LSN on the page
at once; indeed if in this REDO's group there comes another REDO
for the same page it would be wrongly skipped. Instead, we keep
pages pinned, don't change their LSN. When done with all REDOs
of the group we unpin them and stamp them with UNDO's LSN.
- we verify that all log records of a group are about the same table,
for debugging.
mysql-test/r/maria-recovery.result:
result
mysql-test/t/maria-recovery-master.opt:
crash is expected, core file would take room, stack trace would
wake pushbuild up.
mysql-test/t/maria-recovery.test:
Test of recovery from mysql-test (it is already tested as unit tests
in ma_test_recovery) (WL#4137)
- test that, if recovery is made to start on an empty table it can
replay the effects of committed and uncommitted statements (having only
the committed ones in the end result). This should be the first test
for someone writing code of new REDOs.
- test that, if mysqld is crashed and recovery runs we have only
committed statements in the end result. Crashes are done in different
ways: flush nothing (so, uncommitted statement is often missing
from the log => no rollback to do); flush pagecache (implicitely flushes
log (WAL)) and flush log, both causes rollbacks; flush log can also
flush state (state.records etc) to test recovery of the state
(not tested well now as we repair the index anyway).
- test of bug found by Jani and Monty in recovery (two REDO about
the same page in one group).
mysql-test/include/maria_empty_logs.inc:
removes logs, to have a clean sheet for testing recovery.
mysql-test/include/maria_make_snapshot.inc:
copies a table to another directory, or back, or compares both
(comparison is not implemented as physical comparison is impossible
if an UNDO phase happened).
mysql-test/include/maria_make_snapshot_for_comparison.inc:
copies tables to another directory so that they can later
serve as a comparison reference (they are the good tables,
recovery should produce similar ones).
mysql-test/include/maria_make_snapshot_for_feeding_recovery.inc:
When we want to force recovery to start on old tables, we prepare
old tables with this script: we put them in a spare directory.
They are later copied back over mysqltest tables while mysqld is dead.
We also need to copy back the control file, otherwise mysqld,
in recovery, would start from the latest checkpoint: latest
checkpoint plus old tables is not a recovery-possible scenario of course.
mysql-test/include/maria_verify_recovery.inc:
causes mysqld to crash, restores old tables if requested,
lets recovery run, compares resulting tables with reference tables
by using CHECKSUM TABLE.
We don't do any sanity checks on page's LSN in resulting tables, yet.
