Now ma_test1 -M -T and ma_test2 -M -T produces readable, applyable logs
Note: The .MAD file is not binary identical after applying redo compare to a an original file.
(This is becasue we don't have full information which function called PURGE_REDO_BLOCKS).
To verify if a file was correctly applied, we now instead compare row checksums
BitKeeper/etc/ignore:
added storage/maria/tmp/*
include/maria.h:
Added maria_commit() and maria_begin() to be used with external tests
storage/maria/ha_maria.cc:
Ensure maria_def. is read in C mode
storage/maria/ma_blockrec.c:
Fixed redo handling.
_ma_apply_redo_purge_blocks() updated to handle any number of purged blocks
Removed code to make data file idenitcal after redo (can't easily be done). See changeset comments
Now ma_test1 -M -T and ma_test2 -M -T produces readable, applyable logs
storage/maria/ma_commit.c:
More DBUG statements
Moved variable declaration to start of function (portability fix)
Added helper functions 'maria_commit()' and 'maria_begin()'
storage/maria/ma_loghandler.c:
Fixed wrong REDO_PURGE_BLOCKS initialization
storage/maria/ma_recovery.c:
Added UNDO_ROW_UPDATE
Removed wrong setting of lsn (there was no lsn at the used position)
Fixed REDO_PURGE_BLOCKS to handle any number of blocks
storage/maria/ma_test1.c:
Added transaction support (via maria_begin() & maria_commit()) to get a log that can be applied with maria_read_log
storage/maria/ma_test2.c:
Added transaction support (via maria_begin() & maria_commit()) to get a log that can be applied with maria_read_log
storage/maria/ma_test_recovery:
Create temporary files in maria/tmp
Verify files with checksums instead of byte comparisons
storage/maria/maria_chk.c:
When using with -dss we only get filename, records and checksum.
This is useful to do a quick comparision if a files is identical to another one.
storage/maria/maria_def.h:
Added ma_commit()
storage/maria/maria_read_log.c:
Added --help
storage/maria/ma_checkpoint.c:
Definitions of LSN should be collected in
the one file (ma_loghandler_lsn.h)
storage/maria/ma_loghandler.c:
New calls to get first theoretical and first real LSN.
storage/maria/ma_loghandler.h:
New calls to get first theoretical and first real LSN.
storage/maria/ma_loghandler_lsn.h:
Defined yet another impossible LSN to indicate error.
storage/maria/ma_recovery.c:
The first LSN call changed.
storage/maria/maria_read_log.c:
The first LSN call changed.
storage/maria/unittest/Makefile.am:
New unittest added.
storage/maria/unittest/ma_test_loghandler_first_lsn-t.c:
New BitKeeper file ``storage/maria/unittest/ma_test_loghandler_first_lsn-t.c''
errors when reading a log record which has a 0-length header
(like LOGREC_REDO_DROP_TABLE).
storage/maria/ma_loghandler.c:
Functions reading record's header now don't use 0 to indicate error,
as some valid records have a 0-length header (like REDO_DROP_TABLE).
Instead, negative values are used for EOF and error.
storage/maria/ma_loghandler.h:
functions to read record's header now return an int
(either the length of this header (>=0) or some negative values
for EOF or error).
storage/maria/ma_recovery.c:
update to the new log handler's behaviour. Note the @todo.
storage/maria/maria_read_log.c:
inform when program failed
storage/maria/unittest/ma_test_loghandler-t.c:
update to new log handler's API
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
update to new log handler's API
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
update to new log handler's API
found in pushbuild on sapsrv1. Some not fixed as not repeatable
on my machine (32/64 bit issue?).
Fixes for some test failures:
- "maria-connect" now passes;
- "maria": after fixing the obvious reasons for failures, the test
went further and hit a more complex issues: difference in the output
of EXPLAIN output; not fixed;
- "ps_maria" still crashes in assertion
mysqld: ha_maria.cc:1627: virtual int ha_maria::index_read(uchar*, const uchar*, uint, ha_rkey_function): Ass
ertion `inited == INDEX' failed, as already observable in pushbuild.
All this might just be due to an incomplete merge of MyISAM changes
into Maria when 5.1 was last merged to mysql-maria.
include/my_global.h:
temporary fix until next merge of 5.1; without this it does not build
mysql-test/r/maria-connect.result:
position changed
mysql-test/t/maria-connect.test:
If one wants to use the binlog it has to ask for it.
1582 is not used for dup entry error anymore (it was in older 5.1).
Size of first event in binlog was increased by 4 (when the new type
of event "gap" was added).
mysql-test/t/maria.test:
1582 not used anymore in this case
storage/maria/ha_maria.cc:
engine now has to say what binlogging it supports
storage/maria/ma_blockrec.c:
fix for compiler warnings ("comparison is always true" or "always
false")
storage/maria/ma_loghandler.c:
fix for compiler warnings (comparing char* to uchar*)
storage/maria/ma_packrec.c:
fix for compiler warning (fix simply merged from MyISAM)
storage/maria/ma_pagecache.c:
info_check_pin() was not used so gave a compiler warning.
storage/maria/ma_pagecache.h:
fixing typo from the last 5.1->maria merge.
storage/maria/ma_recovery.c:
my_free() has a void* argument, so why cast. byte->uchar.
storage/maria/ma_search.c:
fix for compiler warning (fix simply merged from MyISAM)
storage/maria/maria_read_log.c:
gptr->uchar*
storage/maria/trnman.c:
probable fix for warning found in pushbuild (but not on my machine):
storage/maria/trnman.c: 142
passing argument 6 of \u2018lf_hash_init\u2019 from incompatible pointer type
on sapsrv1.
Unit test for recovery: runs ma_test1 and ma_test2 (both only with
INSERTs and DELETEs; UPDATEs disabled as not handled by recovery)
then moves the tables elswhere; recreates tables from the log, and
compares and fails if there is a difference. Passes now.
Most of maria_read_log.c moved to ma_recovery.c, as it will be re-used
for recovery-from-ha_maria.
Bugfixes of applying of REDO_INSERT, REDO_PURGE_ROW.
Applying of REDO_PURGE_BLOCKS, REDO_DELETE_ALL, REDO_DROP_TABLE,
UNDO_ROW_INSERT (in REDO phase only, i.e. just doing records++),
UNDO_ROW_DELETE, UNDO_ROW_PURGE.
Code cleanups.
Monty: please look for "QQ". Sanja: please look for "Sanja".
Future tasks: recovery of the bitmap (easy), recovery of the state
(make it idempotent), more REDOs (Monty to work on
REDO_UPDATE?), UNDO phase...
Pushing this cset as it looks safe, contains test and bugfixes which
will help Monty implement applying of REDO_UPDATE.
sql/handler.cc:
typo
storage/maria/Makefile.am:
Adding ma_test_recovery (which ma_test_all invokes, and which can
also be run alone). Most of maria_read_log.c moved to ma_recovery.c
storage/maria/ha_maria.cc:
comments
storage/maria/ma_bitmap.c:
fixing comments. 2 -> sizeof(maria_bitmap_marker).
Bitmap-related part of _ma_initialize_datafile() moves in bitmap module.
Now putting the "bm" signature when creating the first bitmap page
(it used to happen only at next open, but that
caused an annoying difference when testing Recovery if the original
run didn't open the table, and it looks more
logical like this: it goes to disk only with its signature correct);
see the "QQ" comment towards the _ma_initialize_data_file() call
in ma_create.c for more).
When reading a bitmap page, verify its signature (happens when normally
using the table or when CHECKing it; not when REPAIRing it).
storage/maria/ma_blockrec.c:
* no need to sync the data file if table is not transactional
* Comments, code cleanup (log-related data moved to log-related code
block, int5store->page_store).
* Store the table's short id into LOGREC_UNDO_ROW_PURGE, like we
do for other records (though this record will soon be replaced
with a CLR).
* If "page" is 1 it means the page which extends from byte
page*block_size+1 to (page+1)*block_size (byte number 1 being
the first byte of the file). The last byte of the file is
data_file_length (same convention).
A new page needs to be created if the last byte of the page is
beyond the last byte of the file, i.e.
(page+1)*block_size+1 > data_file_length, so we correct the test
(bug found when testing log applying for ma_test1 -M -T --skip-update).
* update the page's LSN when removing a row from it during
execution of a REDO_PURGE_ROW record (bug found when testing log
applying for ma_test1 -M -T --skip-update).
* applying of REDO_PURGE_BLOCKs (limited to a one-page range for now).
storage/maria/ma_blockrec.h:
new functions. maria_bitmap_marker does not need to be exported.
storage/maria/ma_close.c:
we can always flush the table's state when closing the last instance
of the table. And it is needed for maria_read_log (as it does
not use maria_lock_database()).
storage/maria/ma_control_file.c:
when in Recovery, some assertions should not be used.
storage/maria/ma_control_file.h:
double-inclusion safe
storage/maria/ma_create.c:
during recovery, don't log records. Comments.
Moving the creation of the first bitmap page to ma_bitmap.c
storage/maria/ma_delete_table.c:
during recovery, don't log records. Log the end-zero of the dropped
table's name, so that recovery can use the string in place without
extending it to fit an end zero.
storage/maria/ma_loghandler.c:
* inwrite_rec_hook also needs access to the MARIA_SHARE, like
prewrite_rec_hook. This will be needed to update
share->records_diff (in the upcoming patch "recovery of the state").
* LOG_DESC::record_ends_group changed to an enum.
* LOG_DESC for LOGREC_REDO_PURGE_BLOCKS and LOGREC_UNDO_ROW_PURGE
corrected
* Sanja please see the @todo LOG BUG
* avoiding DBUG_RETURN(func()) as it gives confusing debug traces.
storage/maria/ma_loghandler.h:
- log write hooks called while the log's lock is held (inwrite_rec_hook)
now need the MARIA_SHARE, like prewrite_rec_hook already had
- instead of a bool saying if this record's type ends groups or not,
we refine: it may not end a group, it may end a group, or it may
be a group in itself. Imagine that we had a physical write failure
to a table before we log the UNDO, we still end up in
external_lock(F_UNLCK) and then we log a COMMIT: we don't want
to consider this COMMIT as ending the group of REDOs (don't want
to execute those REDOs during Recovery), that's why we say "COMMIT
is a group in itself, it aborts any previous group". This also
gives one more sanity check in maria_read_log.
storage/maria/ma_recovery.c:
New Recovery code, replacing the old pseudocode.
Most of maria_read_log moved here.
Call-able from ha_maria, but not enabled yet.
Compared to the previous version of maria_read_log, some bugs have
been fixed, debugging output can go to stdout or a disk file (for now
it's useful for me, later it can be changed), execution of
REDO_DROP_TABLE, REDO_DELETE_ALL, REDO_PURGE_BLOCKS has been added. Duplicate code
has been factored into functions. We abort an unfinished group
of records if we see a record which is a group in itself (like COMMIT).
No need for maria_panic() after a bug (which caused tables to not
be closed) was fixed; if there is yet another bug I prefer to see it.
When opening a table for Recovery, set data_file_length
and key_file_length to their real physical value (these are the
easiest state members to restore :). Warn us if the last page
was truncated (but Recovery handles it).
MARIA_SHARE::state::state::records is now partly recovered (not
idempotent, but works if recreating tables from scracth).
When applying a REDO to a page, stamp it with the UNDO's LSN
(current_group_end_lsn), not with the REDO's LSN; it makes
the table more identical to the original table (easier to compare
the two tables in the end).
Big thing missing: some types of REDOs are not handled,
and the UNDO phase does not exist (missing functions to execute UNDOs
to actually rollback). So for now tests are only inserting/deleting
a few 100 rows, closing the table and seeing if the log is applied ok;
it works. UPDATE not handled.
storage/maria/ma_recovery.h:
new functions: ma_recover() for recovery from inside ha_maria;
_ma_apply_log() for maria_read_log (ma_recover() calls _ma_apply_log()).
Btw, we need to not use the word "recover" for REPAIR/maria_chk anymore.
storage/maria/ma_rename.c:
don't write log records during recovery
storage/maria/ma_test2.c:
- fail if maria_info() or other subtests find some wrong information
- new option -g to skip updates.
- init the translog before creating the table, so that log applying
can work.
- in "#if 0" you'll see some fixed bugs (will be removed).
storage/maria/ma_test_all.sh:
cleanup files. Test log applying.
storage/maria/maria_read_log.c:
most of the logic moves to ma_recovery.c to be shared between
maria_read_log and recovery-from-inside-mysqld.
See ma_recovery.c for additional changes made to the moved code.
storage/maria/ma_test_recovery:
unit test for Recovery. Tests insert and delete,
REDO_UPDATE not yet coded.
Script is called from ma_test_all. Can run standalone.
- LOGREC_REDO_INSERT_ROW_HEAD
- LOGREC_REDO_INSERT_ROW_TAIL
- LOGREC_REDO_PURGE_ROW_HEAD
- LOGREC_REDO_PURGE_ROW_TAIL
sql/sql_yacc.yy:
Fixed typo in previous push
storage/maria/ma_bitmap.c:
Ensure we flush the new bitmap on close
storage/maria/ma_blockrec.c:
Implement applying of REDO entries for
- LOGREC_REDO_INSERT_ROW_HEAD
- LOGREC_REDO_INSERT_ROW_TAIL
- LOGREC_REDO_PURGE_ROW_HEAD
- LOGREC_REDO_PURGE_ROW_TAIL
Split some functions into subfunctions to be able to reuse code
storage/maria/ma_blockrec.h:
Added prototypes for REDO applying functions
storage/maria/ma_loghandler.h:
Safety fix
storage/maria/ma_loghandler_lsn.h:
Avoid compiler warnings
storage/maria/maria_read_log.c:
Added hocks for:
- REDO_INSERT_ROW_HEAD
- REDO_INSERT_ROW_TAIL
- REDO_PURGE_ROW_HEAD
- REDO_PURGE_ROW_TAIL
Added dummy hooks for:
- UNDO_ROW_INSERT
- UNDO_ROW_DELETE
Changed to use maria_pagecache instead of own pagecache (fixed problem with unitialized share->pagecache)
Use maria_panic() at end to ensure that all files are closed properly.
Fixed option handling for --debug
* Don't modify share->base.born_transactional; now it is a value carved
in stone at creation time. share->now_transactional is what can be
modified: it starts at born_transactional, can become false during
ALTER TABLE (when we want no logging), and restored later.
* Not resetting create_rename_lsn to 0 during delete_all or repair.
* when we temporarily disable transactionality, we also change
the page type to PAGECACHE_PLAIN_PAGE: it bypasses some work in the
page cache (optimization), and avoids assertions related to LSNs.
* Disable INSERT DELAYED for transactional tables, because
durability could not be guaranteed (insertion may even not happen)
mysys/mf_keycache.c:
comment
storage/maria/ha_maria.cc:
* a transactional table cannot do INSERT DELAYED
* ha_maria::save_transactional not needed anymore, as now instead
we don't modify MARIA_SHARE::MARIA_BASE_INFO::born_transactional
(born_transactional plays the role of save_transactional), and modify
MARIA_SHARE::now_transactional.
* REPAIR_TABLE log record is now logged by maria_repair()
* comment why we rely on born_transactional to know if we should
skipping a transaction.
* putting together two if()s which test for F_UNLCK
storage/maria/ha_maria.h:
ha_maria::save_transactional not needed anymore (moved to the C layer)
storage/maria/ma_blockrec.c:
* For the block record's code (writing/updating/deleting records),
all that counts is now_transactional, not born_transactional.
* As we now set the page type to PAGECACHE_PLAIN_PAGE for tables
which have now_transactional==FALSE, pagecache will not expect
a meaningful LSN for them in pagecache_unlock_by_link(), so
we can pass it LSN_IMPOSSIBLE.
storage/maria/ma_check.c:
* writing LOGREC_REPAIR_TABLE moves from ha_maria::repair()
to maria_repair(), sounds cleaner (less functions to export).
* when opening a table during REPAIR, don't use the realpath-ed name,
as this may fail if the table has symlinked files (maria_open()
would try to find the data and index file in the directory
of unique_file_name, it would fail if data and index files are in
different dirs); use the unresolved name, open_file_name, which is
the argument which was passed to the maria_open() which created 'info'.
storage/maria/ma_close.c:
assert that when a statement is done with a table, it cleans up
storage/maria/ma_create.c:
new name
storage/maria/ma_delete_all.c:
* using now_transactional
* no reason to reset create_rename_lsn during delete_all (a bug);
also no reason to do it during repair: it was put there because
a positive create_rename_lsn caused a call to check_and_set_lsn()
which asserted in DBUG_ASSERT(block->type == PAGECACHE_LSN_PAGE);
first solution was to use LSN_IMPOSSIBLE in _ma_unpin_all_pages() if
not transactional; but then in the case of ALTER TABLE, with
transactionality temporarily disabled, it asserted in
DBUG_ASSERT(LSN_VALID(lsn)) in pagecache_fwrite() (PAGECACHE_LSN_PAGE
page with zero LSN - bad). The additional solution is to use
PAGECACHE_PLAIN_PAGE when we disable transactionality temporarily: this
avoids checks on the LSN, and also bypasses (optimization) the "flush
log up to LSN" call when the pagecache flushes our page (in other
words, no WAL needed).
storage/maria/ma_delete_table.c:
use now_transactional
storage/maria/ma_locking.c:
assert that when a statement is done with a table, it cleans up.
storage/maria/ma_loghandler.c:
* now_transactional should be used to test if we want a log record.
* Assertions to make sure dummy_transaction_object is not spoilt
by its many users.
storage/maria/ma_open.c:
base.transactional -> base.born_transactional
storage/maria/ma_pagecache.c:
missing name for page's type. Comment for future.
storage/maria/ma_rename.c:
use now_transactional
storage/maria/maria_chk.c:
use born_transactional
storage/maria/maria_def.h:
MARIA_BASE_INFO::transactional renamed to born_transactional.
MARIA_SHARE::now_transactional introduced.
_ma_repair_write_log_record() is made local to ma_check.c.
Macros to temporarily disable, and re-enable, transactionality for a
table.
storage/maria/maria_read_log.c:
assertions and using the new macros. Adding a forgotten resetting
when we finally close all tables.
BitKeeper/etc/ignore:
added storage/maria/maria_read_log
support-files/compiler_warnings.supp:
Ignore function used when debugging (can be called from gdb)
comments; remember the UNDO's LSN for storing it in pages when
executing REDO's (to imitate what the runtime code does)
storage/maria/maria_read_log.c:
comments; remember the UNDO's LSN for storing it in pages when
executing REDO's (to imitate what the runtime code does)
fixes for build failures; copyrights; small bugfixes and comments
mysys/Makefile.am:
missing .h breaks building from tarball
storage/maria/ma_loghandler.c:
applying Serg's bugfix of trnman_new_trid() to translog_assign_id_to_share()
storage/maria/ma_loghandler.h:
copyright
storage/maria/ma_loghandler_lsn.h:
copyright
storage/maria/maria_read_log.c:
fix for compiler warnings. Comments.
Close tables when program ends.
storage/maria/maria_read_log.c:
assertions to protect against future bugs (especially, to ensure
that replaying DROP TABLE, if implemented, wouldn't leave open tables
behind it)
- new program maria_read_log to display and apply log records
found in a Maria log (see file's revision comment)
- minor, misc fixes
storage/maria/Makefile.am:
new program maria_read_log
storage/maria/ha_maria.cc:
create control file if missing
storage/maria/ma_blockrec.c:
0 -> LSN_IMPOSSIBLE; comments
storage/maria/ma_checkpoint.h:
preparations for Checkpoint module
storage/maria/ma_close.c:
comment
storage/maria/ma_control_file.c:
renaming constants.
Possibility to say "open control file but don't create it if it's
missing" (used by maria_read_log which does not want to create
anything)
storage/maria/ma_control_file.h:
renaming constants
storage/maria/ma_create.c:
I had duplicated "linkname" and "linkname_ptr", now I see it's not
needed, reverting. Indeed those variables don't contain interesting
information; fixing log record accordingly (the links are in
ci->data/index_file_name). Storing keystart in log record is needed,
to know at which size we must extend the file if we replay
LOGREC_CREATE_TABLE.
storage/maria/ma_loghandler.c:
some structures need to be known to maria_read_log.c, taking
them to ma_loghandler.h
storage/maria/ma_loghandler.h:
we have page_store, adding page_korr.
translog_lock() made public, because Checkpoint will need it (to
write to control file).
Some structures moved from ma_loghandler.c because maria_read_log.c
needs them (needs to know the execute-in-REDO-phase hooks of each
record).
storage/maria/ma_loghandler_lsn.h:
constants defined in ma_control_file.h serve everywhere,
and they relate to LSNs, so putting them in ma_loghandler_lsn.h.
Stronger constraints in LSN_VALID().
storage/maria/ma_pagecache.c:
renaming constants
storage/maria/ma_recovery.h:
copyright
storage/maria/ma_test1.c:
new prototype
storage/maria/ma_test2.c:
new prototype
storage/maria/trnman_public.h:
double-inclusion safe
storage/maria/unittest/ma_control_file-t.c:
constants renamed, new prototype
storage/maria/unittest/ma_test_loghandler-t.c:
constants renamed, new prototype
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
constants renamed, new prototype
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
constants renamed, new prototype
storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
constants renamed, new prototype
storage/myisam/mi_close.c:
comment
storage/maria/maria_read_log.c:
program to read and print log records from a Maria transaction log,
and optionally apply them to tables. Very basic, early version.
Should serve as a base for Recovery's code. Designed to be idempotent.
Create a log by running maria.test, then cd to var/master-data
and run "maria_read_log --only-display" to see info about records;
run "maria_read_log --display-and-apply" to also apply the records
to tables (it's more interesting if you first wipe out the
tables in var/master-data/test, to see how they get re-created).
Only a few records are handled by now: LONG_TRANSACTION_ID,
COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for
REDO_INSERT_ROW_HEAD where I could use Monty's help (search for
"Monty" in the file). Note: changes to the index pages, index's header
and bitmap pages are not properly logged yet, so don't expect
the program to work with that.
