12 commits

Author	SHA1	Message	Date
unknown	0f1feefa03	WL#3071 Maria checkpoint ... Ability for flush_pagecache_blocks() to flush only certain pages of a file, as instructed by an option "filter" pointer-to-function argument; Checkpoint and background dirty page flushing use that to flush only pages which have been dirty for long enough and bitmap pages. Fix for a bug in flush_cached_blocks() (no idea if it could produce a bug in real life, but theoretically it is). Testing checkpoint in ma_test_recovery via ma_test1 and ma_test2. Background checkpoint & dirty pages flush thread is still disabled by default in ha_maria. mysql-test/r/maria.result: result update storage/maria/ha_maria.cc: blank after function comment storage/maria/ma_checkpoint.c: Using an enum instead of 0/1/2 (applying Sanja's review comments). The comment about "this is an horizon" can be removed as Sanja created translog_next_LSN() which parse_checkpoint_record() uses. Variables in ma_checkpoint_background() cannot be declared in the for() as their value must not be reset at each iteration! storage/maria/ma_pagecache.c: adding to flush_pagecache_blocks() optional arguments 'filter' (pointer to function) and 'filter_arg'; if filter!=NULL this function will be called for each block of the file and will reply if this block and following ones should be flushed or not (3 possible replies). Fixing a bug when flush_cached_blocks() skips a pinned page: it has to unset PCBLOCK_IN_FLUSH set by flush_pagecache_blocks_int(). storage/maria/ma_pagecache.h: flush_pagecache_blocks() is changed to take "filter" and "filter_arg" arguments. "filter", if it is not NULL, may return one value among enum pagecache_flush_filter_result. storage/maria/ma_recovery.c: open_count=0 when closing tables at the end of recovery. storage/maria/ma_test1.c: Optional checkpoints (-H#) at various stages (stages similar to --testflag), for testing of checkpoints. storage/maria/ma_test2.c: Optional checkpoints (-H#) at various stages (stages similar to -t), for testing of checkpoints. storage/maria/ma_test_recovery.expected: Result update: the results of the additional test run with -H# (checkpoints) are added here. They are exactly identical to without checkpoints except that the index's Root (printed by maria_chk) is more correct when using checkpoints. This is because checkpoint flushed the state, so it happens to be correct, while no-checkpoint does not flush the state, and recovery does not recover indexes so Root is never fixed. When we recover indices, this will go away. storage/maria/ma_test_recovery: We duplicate the loop of tests to add an additional run with checkpoints at various stages, to see if maria_read_log uses them fine.	2007-10-17 16:55:26 +02:00
unknown	8b5dddbc00	WL#3072 Maria recovery ... Progress reports on stderr if doing recovery from ha_maria; don't do checkpoints if activity since last checkpoint < 2MB (no change in fact as background thread is disabled for now); recovery trace is only if EXTRA_DEBUG now (better for benchmarks). storage/maria/ma_checkpoint.c: don't do checkpoints if activity (log writes plus page flushes) since last checkpoint was < 2MB. storage/maria/ma_recovery.c: progress reports in recovery (10%, transactions left to rollback etc); that is only if from ha_maria and is displayed on stderr. Recovery trace is now created only if EXTRA_DEBUG. storage/maria/ma_test_recovery.expected: update (--debug gone) storage/maria/ma_test_recovery: don't use --debug, as it can absent from binary	2007-09-25 11:54:35 +02:00
unknown	cec8ac3e07	WL#3071 Maria checkpoint ... Finally this is the real checkpoint code. It however exhibits unstabilities when a checkpoint runs concurrently with data-modifying clients (table corruption, transaction log's assertions) so for now a checkpoint is taken only at startup after recovery and at shutdown, i.e. not in concurrent situations. Later we will let it run periodically, as well as flush dirty pages periodically (almost all needed code is there already, only pagecache code is written but not committed). WL#3072 Maria recovery * replacing UNDO_ROW_PURGE with CLR_END; testing of those CLR_END via ma_test2 which has INSERTs failing with duplicate keys. * replaying of REDO_RENAME_TABLE Now, off to test Recovery in ha_maria :) BitKeeper/deleted/.del-ma_least_recently_dirtied.c: Delete: storage/maria/ma_least_recently_dirtied.c BitKeeper/deleted/.del-ma_least_recently_dirtied.h: Delete: storage/maria/ma_least_recently_dirtied.h storage/maria/Makefile.am: compile Checkpoint module storage/maria/ha_maria.cc: When ha_maria starts, do a recovery from last checkpoint. Take a checkpoint when that recovery has ended and when ha_maria shuts down cleanly. storage/maria/ma_blockrec.c: * even if my_sync() fails we have to my_close() (otherwise we leak a descriptor) * UNDO_ROW_PURGE is replaced by a simple CLR_END for UNDO_ROW_INSERT, as promised in the old comment; it gives us skipping during the UNDO phase. storage/maria/ma_check.c: All REDOs before create_rename_lsn are ignored by Recovery. So create_rename_lsn must be set only after all data/index has been flushed and forced to disk. We thus move write_log_record_for_repair() to after _ma_flush_tables_files_after_repair(). storage/maria/ma_checkpoint.c: Checkpoint module. storage/maria/ma_checkpoint.h: optional argument if caller wants a thread to periodically take checkpoints and flush dirty pages. storage/maria/ma_create.c: * no need to init some vars as the initial bzero(share) takes care of this. * update to new function's name * even if we fail in my_sync() we have to my_close() storage/maria/ma_extra.c: Checkpoint reads share->last_version under intern_lock, so we make maria_extra() update it under intern_lock. THR_LOCK_maria still needed because of _ma_test_if_reopen(). storage/maria/ma_init.c: destroy checkpoint module when Maria shuts down. storage/maria/ma_loghandler.c: * UNDO_ROW_PURGE gone (see ma_blockrec.c) * we need to remember the LSN of the LOGREC_FILE_ID for a share, because this LSN is needed into the checkpoint record (Recovery wants to know the validity domain of an id->name mapping) * translog_get_horizon_no_lock() needed for Checkpoint * comment about failing assertion (Sanja knows) * translog_init_reader_data() thought that translog_read_record_header_scan() returns 0 in case of error, but 0 just means "0-length header". * translog_assign_id_to_share() now needs the MARIA_HA because LOGREC_FILE_ID uses a log-write hook. * Verify that (de)assignment of share->id happens only under intern_lock, as Checkpoint reads this id with intern_lock. * translog_purge() can accept TRANSLOG_ADDRESS, not necessarily a real LSN. storage/maria/ma_loghandler.h: prototype updates storage/maria/ma_open.c: no need to initialize "res" storage/maria/ma_pagecache.c: When taking a checkpoint, we don't need to know the maximum rec_lsn of dirty pages; this LSN was intended to be used in the two-checkpoint rule, but last_checkpoint_lsn is as good. 4 bytes for stored_list_size is enough as PAGECACHE::blocks (number of blocks which the pagecache can contain) is int. storage/maria/ma_pagecache.h: new prototype storage/maria/ma_recovery.c: * added replaying of REDO_RENAME_TABLE * UNDO_ROW_PURGE gone (see ma_blockrec.c), replaced by CLR_END * Recovery from the last checkpoint record now possible * In new_table() we skip the table if the id->name mapping is older than create_rename_lsn (mapping dates from lsn_of_file_id). * in get_MARIA_HA_from_REDO_record() we skip the record if the id->name mapping is newer than the record (can happen if processing a record which is before the checkpoint record). * parse_checkpoint_record() has to return a LSN, that's what caller expects storage/maria/ma_rename.c: new function's name; log end zeroes of tables' names (ease recovery) storage/maria/ma_test2.c: * equivalent of ma_test1's --test-undo added (named -u here). * -t=1 now stops right after creating the table, so that we can test undoing of INSERTs with duplicate keys (which tests the CLR_END logged by _ma_write_abort_block_record()). storage/maria/ma_test_recovery.expected: Result of testing undoing of INSERTs with duplicate keys; there are some differences in maria_chk -dvv but they are normal (removing records does not shrink data/index file, does not put back the "analyzed, optimized keys"(etc) index state. storage/maria/ma_test_recovery: Test undoing of INSERTs with duplicate keys, using ma_test2; when such INSERT happens, it logs REDO_INSERT, UNDO_INSERT, REDO_DELETE, CLR_END; we abort after that, and test that CLR_END causes recovery to jump over UNDO_INSERT. storage/maria/ma_write.c: comment storage/maria/maria_chk.c: comment storage/maria/maria_def.h: * a new bit in MARIA_SHARE::in_checkpoint, used to build a list of unique shares during Checkpoint. * MARIA_SHARE::lsn_of_file_id added: the LSN of the last LOGREC_FILE_ID for this share; needed to know to which LSN domain the mappings found in the Checkpoint record apply (new mappings should not apply to old REDOs). storage/maria/trnman.c: * small changes to how trnman_collect_transactions() fills its buffer; it also uses a non-dummy lsn_read_non_atomic() found in ma_checkpoint.h	2007-09-12 11:27:34 +02:00
unknown	0b2ba820c3	WL#3072 Maria recovery ... * testing of execution of UNDO_ROW_UPDATE * when executing an UNDO_ROW_UPDATE, store "UNDO_ROW_UPDATE" as "type of undone record" into the CLR_END record. storage/maria/ma_blockrec.c: When logging a CLR_END in write_block_record(), it can be for a DELETE or for an UPDATE (now that Monty has coded execution of UNDO_UPDATE) storage/maria/ma_loghandler.c: UNDO_ROW_UPDATE's execution coded, so no crash storage/maria/ma_recovery.c: UNDO_ROW_UPDATE's execution now coded, so no crash storage/maria/ma_test1.c: upper case letter storage/maria/ma_test_recovery.expected: output of testing execution of UNDO_ROW_UPDATE. Table's checksum not recovered (known issue not specific to UPDATE). storage/maria/ma_test_recovery: Test execution of UNDO_ROW_UPDATE: first we stop ma_test1 after deletes and commit, then we stop ma_test1 after updates and abort; we verify that updates are rolled back by comparing tables	2007-09-11 11:11:22 +02:00
unknown	2291f932b2	- WL#3072 Maria Recovery: ... Recovery of state.records (the count of records which is stored into the header of the index file). For that, state.is_of_lsn is introduced; logic is explained in ma_recovery.c (look for "Recovery of the state"). The net gain is that in case of crash, we now recover state.records, and it is idempotent (ma_test_recovery tests it). state.checksum is not recovered yet, mail sent for discussion. - WL#3071 Maria Checkpoint: preparation for it, by protecting all modifications of the state in memory or on disk with intern_lock (with the exception of the really-often-modified state.records, which is now protected with the log's lock, see ma_recovery.c (look for "Recovery of the state"). Also, if maria_close() sees that Checkpoint is looking at this table it will not my_free() the share. - don't compute row's checksum twice in case of UPDATE (correction to a bugfix I made yesterday). storage/maria/ha_maria.cc: protect state write with intern_lock (against Checkpoint) storage/maria/ma_blockrec.c: * don't reset trn->rec_lsn in _ma_unpin_all_pages(), because it should wait until we have corrected the allocation in the bitmap (as the REDO can serve to correct the allocation during Recovery); introducing _ma_finalize_row() for that. * In a changeset yesterday I moved computation of the checksum into write_block_record(), to fix a bug in UPDATE. Now I notice that maria_update() already computes the checksum, it's just that it puts it into info->cur_row while _ma_update_block_record() uses info->new_row; so, removing the checksum computation from write_block_record(), putting it back into allocate_and_write_block_record() (which is called only by INSERT and UNDO_DELETE), and copying cur_row->checksum into new_row->checksum in _ma_update_block_record(). storage/maria/ma_check.c: new prototypes, they will take intern_lock when writing the state; also take intern_lock when changing share->kfile. In both cases this is to protect against Checkpoint reading/writing the state or reading kfile at the same time. Not updating create_rename_lsn directly at end of write_log_record_for_repair() as it wouldn't have intern_lock. storage/maria/ma_close.c: Checkpoint builds a list of shares (under THR_LOCK_maria), then it handles each such share (under intern_lock) (doing flushing etc); if maria_close() freed this share between the two, Checkpoint would see a bad pointer. To avoid this, when building the list Checkpoint marks each share, so that maria_close() knows it should not free it and Checkpoint will free it itself. Extending the zone covered by intern_lock to protect against Checkpoint reading kfile, writing state. storage/maria/ma_create.c: When we update create_rename_lsn, we also update is_of_lsn to the same value: it is logical, and allows us to test in maria_open() that the former is not bigger than the latter (the contrary is a sign of index header corruption, or severe logging bug which hinders Recovery, table needs a repair). _ma_update_create_rename_lsn_on_disk() also writes is_of_lsn; it now operates under intern_lock (protect against Checkpoint), a shortcut function is available for cases where acquiring intern_lock is not needed (table's creation or first open). storage/maria/ma_delete.c: if table is transactional, "records" is already decremented when logging UNDO_ROW_DELETE. storage/maria/ma_delete_all.c: comments storage/maria/ma_extra.c: Protect modifications of the state, in memory and/or on disk, with intern_lock, against a concurrent Checkpoint. When state goes to disk, update it's is_of_lsn (by calling the new _ma_state_info_write()). In HA_EXTRA_FORCE_REOPEN, don't set share->changed to 0 (undoing a change I made a few days ago) and ASK_MONTY storage/maria/ma_locking.c: no real code change here. storage/maria/ma_loghandler.c: Log-write-hooks for updating "state.records" under log's mutex when writing/updating/deleting a row or deleting all rows. storage/maria/ma_loghandler_lsn.h: merge (make LSN_ERROR and LSN_REPAIRED_BY_MARIA_CHK different) storage/maria/ma_open.c: When opening a table verify that is_of_lsn >= create_rename_lsn; if false the header must be corrupted. _ma_state_info_write() is split in two: _ma_state_info_write_sub() which is the old _ma_state_info_write(), and _ma_state_info_write() which additionally takes intern_lock if requested (to protect against Checkpoint) and updates is_of_lsn. _ma_open_keyfile() should change kfile.file under intern_lock to protect Checkpoint from reading a wrong kfile.file. storage/maria/ma_recovery.c: Recovery of state.records: when the REDO phase sees UNDO_ROW_INSERT which has a LSN > state.is_of_lsn it increments state.records. Same for UNDO_ROW_DELETE and UNDO_ROW_PURGE. When closing a table during Recovery, we know its state is at least as new as the current log record we are looking at, so increase is_of_lsn to the LSN of the current log record. storage/maria/ma_rename.c: update for new behaviour of _ma_update_create_rename_lsn_on_disk(). storage/maria/ma_test1.c: update to new prototype storage/maria/ma_test2.c: update to new prototype (actually prototype was changed days ago, but compiler does not complain about the extra argument??) storage/maria/ma_test_recovery.expected: new result file of ma_test_recovery. Improvements: record count read from index's header is now always correct. storage/maria/ma_test_recovery: "rm" fails if file does not exist. Redirect stderr of script. storage/maria/ma_write.c: if table is transactional, "records" is already incremented when logging UNDO_ROW_INSERT. Comments. storage/maria/maria_chk.c: update is_of_lsn too storage/maria/maria_def.h: - MARIA_STATE_INFO::is_of_lsn which is used by Recovery. It is stored into the index file's header. - Checkpoint can now mark a table as "don't free this", and maria_close() can reply "ok then you will free it". - new functions storage/maria/maria_pack.c: update for new name	2007-09-07 15:02:30 +02:00
unknown	d53991853e	- speed optimization: ... minimize writes to transactional Maria tables: don't write data pages, state, and open_count at the end of each statement. Data pages will be written by a background thread periodically. State will be written by Checkpoint periodically. open_count serves to detect when a table is potentially damaged due to an unclean mysqld stop, but thanks to recovery an unclean mysqld stop will be corrected and so open_count becomes useless. As state is written less often, it is often obsolete on disk, we thus should avoid to read it from disk. - by removing the data page writes above, it is necessary to put it back at the start of some statements like check, repair and delete_all. It was already necessary in fact (see ma_delete_all.c). - disabling CACHE INDEX on Maria tables for now (fixes crash of test 'key_cache' when run with --default-storage-engine=maria). - correcting some fishy code in maria_extra.c (we possibly could lose index pages when doing a DROP TABLE under Windows, in theory). storage/maria/ha_maria.cc: disable CACHE INDEX in Maria for now (there is a single cache for now), it crashes and it's not a priority storage/maria/ma_bitmap.c: debug message storage/maria/ma_check.c: The statement before maria_repair() may not flush state, so it needs to be done by maria_repair() (indeed this function uses maria_open(HA_OPEN_COPY) so reads state from disk, so needs to find it up-to-date on disk). For safety (but normally this is not needed) we remove index blocks out of the cache before repairing. _ma_flush_blocks() becomes _ma_flush_table_files_after_repair(): it now additionally flushes the data file and state and syncs files. As a side effect, the assertion "no WRITE_CACHE_USED" from _ma_flush_table_files() fired so we move all end_io_cache() done at the end of repair to before the calls to _ma_flush_table_files_after_repair(). storage/maria/ma_close.c: when closing a transactional table, we fsync it. But we need to do this only after writing its state. We need to write the state at close time only for transactional tables (the other tables do that at last unlock). Putting back the O_RDONLY||crashed condition which I had removed earlier. Unmap the file before syncing it (does not matter now as Maria does not use mmap) storage/maria/ma_delete_all.c: need to flush data pages before chsize-ing it. Was needed even when we flushed data pages at the end of each statement, because we didn't anyway do it if under LOCK TABLES: the change here thus fixes this bug: create table t(a int) engine=maria;lock tables t write; insert into t values(1);delete from t;unlock tables;check table t; "Size of datafile is: 16384 Should be: 8192" (an obsolete page went to disk after the chsize(), at unlock time). storage/maria/ma_extra.c: When doing share->last_version=0, we make the MARIA_SHARE-in-memory invisible to future openers, so need to have an up-to-date state on disk for them. The same way, future openers will reopen the data and index file, so they will not find our cached blocks, so we need to flush them to disk. In HA_EXTRA_FORCE_REOPEN, this probably happens naturally as all tables normally get closed, we however add a safety flush. In HA_EXTRA_PREPARE_FOR_RENAME, we need to do the flushing. On Windows we additionally need to close files. In HA_EXTRA_PREPARE_FOR_DROP, we don't need to flush anything but remove dirty cached blocks from memory. On Windows we need to close files. Closing files forces us to sync them before (requirement for transactional tables). For mutex reasons (don't lock intern_lock twice), we move maria_lock_database() and _ma_decrement_open_count() first in the list of operations. Flush also data file in HA_EXTRA_FLUSH. storage/maria/ma_locking.c: For transactional tables: - don't write data pages / state at unlock time; as a consequence, "share->changed=0" cannot be done. - don't write state in _ma_writeinfo() - don't maintain open_count on disk (Recovery corrects the table in case of crash anyway, and we gain speed by not writing open_count to disk), For non-transactional tables, flush the state at unlock only if the table was changed (optimization). Code which read the state from disk is relevant only with external locking, we disable it (if want to re-enable it, it shouldn't for transactional tables as state on disk may be obsolete (such tables does not flush state at unlock anymore). The comment "We have to flush the write cache" is now wrong because maria_lock_database(F_UNLCK) now happens before thr_unlock(), and we are not using external locking. storage/maria/ma_open.c: _ma_state_info_read() is only used in ma_open.c, making it static storage/maria/ma_recovery.c: set MARIA_SHARE::changed to TRUE when we are going to apply a REDO/UNDO, so that the state gets flushed at close. storage/maria/ma_test_recovery.expected: Changes introduced by this patch: - good: the "open" (table open, not properly closed) is gone, it was pointless for a recovered table - bad: stemming from different moments of writing the index's state probably (_ma_writeinfo() used to write the state after every row write in ma_test* programs, doesn't anymore as the table is transactional): some differences in indexes (not relevant as we don't yet have recovery for them); some differences in count of records (changed from a wrong value to another wrong value) (not relevant as we don't recover this count correctly yet anyway, though a patch will be pushed soon). storage/maria/ma_test_recovery: for repeatable output, no names of varying directories. storage/maria/maria_chk.c: function renamed storage/maria/maria_def.h: Function became local to ma_open.c. Function renamed.	2007-09-06 16:53:26 +02:00
unknown	ac4ad9bdba	WL#3072 Maria Recovery ... misc fixes of execution of UNDOs in the UNDO phase: - into the CLR_END, store the LSN of the _previous_ UNDO (we debated what was best, so far we're going with "previous"; later we can change to "current" if needed), and store the type of record which is being undone (needed to know how to update state.records when we see the CLR_END during the REDO phase). - declaring all UNDOs and CLR_END as "compressed" - when executing an UNDO in the UNDO phase, state.records is updated as a hook when writing CLR_END (needed for "recovery of the state"), and so is trn->undo_lsn (needed for when we have checkpoints). - bugfix (execution of UNDO_ROW_DELETE didn't store the correct checksum into the re-inserted row, maria_chk -r thus threw the row away). - modifications of ma_test1: where to stop is now driven by --testflag; --test-undo just tells how to stop (flush data, flush log, nothing). - ma_test_recovery: testing of the UNDO phase, more testing of the REDO phase, identification of a bug. storage/maria/ma_blockrec.c: - bugfix: execution of UNDO_ROW_DELETE didn't store the correct checksum into the row (leading to "maria_chk -r" eliminating the re-inserted row, net effect was that rollback appeared to have rolled back no deletion). Reason was that write_block_record() used info->cur_row.checksum, while "row" can be != &info->cur_row (case of UNDO_ROW_DELETE). After fixing this, problems with _ma_update_block_record() appeared; indeed checksum was computed by allocate_and_write_block_record() while _ma_update_block_record() directly calls write_block_record(). Solution is to compute checksum in write_block_record() instead. - when executing an UNDO, we now pass the LSN of the _previous_ UNDO to block_format functions. This LSN can be 0 (if the being-executed UNDO was the transaction's first UNDO), so "undo_lsn==0" cannot work anymore to indicate "this is not UNDO work". Using undo_lsn==LSN_ERROR instead (this is an impossible LSN). - store into CLR_END the type of log record which was undone (INSERT/UPDATE/DELETE); needed for Recovery to know if/how it has to update state.records if it sees this CLR_END in the REDO phase. - when writing the CLR_END in _ma_apply_undo_row_insert(), the place to store file's id is log_data+LSN_STORE_SIZE. - in _ma_apply_undo_row_insert(), the records-- is moved to a hook when writing the CLR_END (this way it is under log's mutex which is needed for "recovery of the state") storage/maria/ma_loghandler.c: - all UNDOs, and CLR_END, start with the LSN of another UNDO; so we can declare them "compressed". - write_hook_for_clr_end() to set trn->undo_lsn (to the previous UNDO's LSN) under log's lock (like UNDOs set trn->undo_lsn under log's lock), and also update, if appropriate, state.records. - reset share->id to 0 when deassigning; not useful for now but sounds logical. storage/maria/ma_recovery.c: - if no table is found for a REDO, it's not an error; for an UNDO, it is - in the REDO phase, when we see a CLR_END we must update trn->undo_lsn and sometimes state.records. - in the UNDO phase, when we execute an UNDO_ROW_INSERT: * update trn->undo_lsn only after executing the record * store the _previous_ undo_lsn into the CLR_END - at the end of the REDO phase, when we recreate TRN objects, they have already their long id in the log (either via a LOGREC_LONG_TRANSACTION_ID, or in a checkpoint record), don't write a new, useless LOGREC_LONG_TRANSACTION_ID for them. storage/maria/ma_test1.c: * where to stop execution is now driven by --testflag and not --test-undo (ma_test2 already has --testflag for the same purpose). This allows us to do a clean stop (with commit) at any point. * --test-undo=# tells how to abort (flush all pages (which implies flushing log) or only log or nothing); all such "ways of crashing" are tested in ma_test_recovery storage/maria/ma_test_recovery: * Testing execution of UNDOs, with and without BLOBs. * Testing idempotency of REDOs. * See @todo for a probable bug with BLOBs. * maria_chk -rq instead of -r, as with -q it nicely stops on any problem in the data file (like the checksum bug see comment of ma_blockrec.c). * Testing if log was written by UNDO phase (often expected), not written by REDO phase (always expected). * Less output on the screen, compares with expected output in the end. * some shell thingies like "set --" and $# are courtesy of Danny and Pekka. storage/maria/maria_read_log.c: when only displaying the records, don't do an UNDO phase storage/maria/ma_test_recovery.expected: This is the expected output of a great part of ma_test_recovery. ma_test_recovery compares its output to the expected output and tells if different. If we look at this file it mentions differences in checksum (normal, it's not recovered yet) and in records count (getting a correct records' count when recovery starts on an already existing table, like when testing rollback, is coded but not yet pushed).	2007-09-06 16:04:36 +02:00
unknown	2cc2f3e757	Generalized the way update and redo extends the size of a directory record. ... storage/maria/ma_blockrec.c: Generalized the way update and redo extends the size of a directory record. This will (for now) ensure that data files are idenitical after normal run and after a apply-log run. storage/maria/ma_open.c: Disabled reservation of transid on rows (for now) as these are not yet used. (I had to disable this as otherwise update thougth rows had grown in size when they hadn't and we had thus different row sizes on update and redo, which caused different block information) storage/maria/ma_test1.c: Added comment storage/maria/ma_test2.c: Do commit on error/abort storage/maria/ma_test_all.sh: Some more testing (to cover a bug that was not found in previous runs) storage/maria/ma_test_recovery: More tests	2007-08-31 10:19:54 +03:00
unknown	f7b766c029	Added maria_commit() and maria_begin() to be used with external tests ... Now ma_test1 -M -T and ma_test2 -M -T produces readable, applyable logs Note: The .MAD file is not binary identical after applying redo compare to a an original file. (This is becasue we don't have full information which function called PURGE_REDO_BLOCKS). To verify if a file was correctly applied, we now instead compare row checksums BitKeeper/etc/ignore: added storage/maria/tmp/* include/maria.h: Added maria_commit() and maria_begin() to be used with external tests storage/maria/ha_maria.cc: Ensure maria_def. is read in C mode storage/maria/ma_blockrec.c: Fixed redo handling. _ma_apply_redo_purge_blocks() updated to handle any number of purged blocks Removed code to make data file idenitcal after redo (can't easily be done). See changeset comments Now ma_test1 -M -T and ma_test2 -M -T produces readable, applyable logs storage/maria/ma_commit.c: More DBUG statements Moved variable declaration to start of function (portability fix) Added helper functions 'maria_commit()' and 'maria_begin()' storage/maria/ma_loghandler.c: Fixed wrong REDO_PURGE_BLOCKS initialization storage/maria/ma_recovery.c: Added UNDO_ROW_UPDATE Removed wrong setting of lsn (there was no lsn at the used position) Fixed REDO_PURGE_BLOCKS to handle any number of blocks storage/maria/ma_test1.c: Added transaction support (via maria_begin() & maria_commit()) to get a log that can be applied with maria_read_log storage/maria/ma_test2.c: Added transaction support (via maria_begin() & maria_commit()) to get a log that can be applied with maria_read_log storage/maria/ma_test_recovery: Create temporary files in maria/tmp Verify files with checksums instead of byte comparisons storage/maria/maria_chk.c: When using with -dss we only get filename, records and checksum. This is useful to do a quick comparision if a files is identical to another one. storage/maria/maria_def.h: Added ma_commit() storage/maria/maria_read_log.c: Added --help	2007-08-29 09:03:10 +03:00
unknown	ec547ae830	fixes of bad merge (probably). Less duplication in ma_test_recovery. ... storage/maria/ma_range.c: old code, which is wrong now (key_len is now a local variable, not initialized). Fixes some problems in running ma_test2. storage/maria/ma_test2.c: keypart_map should be used instead of length (fix of bad merge probably) storage/maria/ma_test_recovery: less duplication in this script (one loop instead).	2007-08-06 16:13:42 +02:00
unknown	9b1e83dba7	porting Serg's fix for BUG#30094 to Maria. Now ma_test_all passes. ... maria.test and ps_maria.test still fail. mysys/mf_keycache.c: split string annoys some compilers storage/maria/ha_maria.cc: fix for compiler warnings storage/maria/ma_test1.c: porting Serg's fix for BUG#30094 to Maria storage/maria/ma_test2.c: porting Serg's fix for BUG#30094 to Maria storage/maria/ma_test3.c: porting Serg's fix for BUG#30094 to Maria storage/maria/ma_test_recovery: don't print ma_test1's messages if no problem	2007-07-27 16:11:40 +02:00
unknown	6bbca54d7d	WL#3072 - Maria recovery ... Unit test for recovery: runs ma_test1 and ma_test2 (both only with INSERTs and DELETEs; UPDATEs disabled as not handled by recovery) then moves the tables elswhere; recreates tables from the log, and compares and fails if there is a difference. Passes now. Most of maria_read_log.c moved to ma_recovery.c, as it will be re-used for recovery-from-ha_maria. Bugfixes of applying of REDO_INSERT, REDO_PURGE_ROW. Applying of REDO_PURGE_BLOCKS, REDO_DELETE_ALL, REDO_DROP_TABLE, UNDO_ROW_INSERT (in REDO phase only, i.e. just doing records++), UNDO_ROW_DELETE, UNDO_ROW_PURGE. Code cleanups. Monty: please look for "QQ". Sanja: please look for "Sanja". Future tasks: recovery of the bitmap (easy), recovery of the state (make it idempotent), more REDOs (Monty to work on REDO_UPDATE?), UNDO phase... Pushing this cset as it looks safe, contains test and bugfixes which will help Monty implement applying of REDO_UPDATE. sql/handler.cc: typo storage/maria/Makefile.am: Adding ma_test_recovery (which ma_test_all invokes, and which can also be run alone). Most of maria_read_log.c moved to ma_recovery.c storage/maria/ha_maria.cc: comments storage/maria/ma_bitmap.c: fixing comments. 2 -> sizeof(maria_bitmap_marker). Bitmap-related part of _ma_initialize_datafile() moves in bitmap module. Now putting the "bm" signature when creating the first bitmap page (it used to happen only at next open, but that caused an annoying difference when testing Recovery if the original run didn't open the table, and it looks more logical like this: it goes to disk only with its signature correct); see the "QQ" comment towards the _ma_initialize_data_file() call in ma_create.c for more). When reading a bitmap page, verify its signature (happens when normally using the table or when CHECKing it; not when REPAIRing it). storage/maria/ma_blockrec.c: * no need to sync the data file if table is not transactional * Comments, code cleanup (log-related data moved to log-related code block, int5store->page_store). * Store the table's short id into LOGREC_UNDO_ROW_PURGE, like we do for other records (though this record will soon be replaced with a CLR). * If "page" is 1 it means the page which extends from byte page*block_size+1 to (page+1)*block_size (byte number 1 being the first byte of the file). The last byte of the file is data_file_length (same convention). A new page needs to be created if the last byte of the page is beyond the last byte of the file, i.e. (page+1)*block_size+1 > data_file_length, so we correct the test (bug found when testing log applying for ma_test1 -M -T --skip-update). * update the page's LSN when removing a row from it during execution of a REDO_PURGE_ROW record (bug found when testing log applying for ma_test1 -M -T --skip-update). * applying of REDO_PURGE_BLOCKs (limited to a one-page range for now). storage/maria/ma_blockrec.h: new functions. maria_bitmap_marker does not need to be exported. storage/maria/ma_close.c: we can always flush the table's state when closing the last instance of the table. And it is needed for maria_read_log (as it does not use maria_lock_database()). storage/maria/ma_control_file.c: when in Recovery, some assertions should not be used. storage/maria/ma_control_file.h: double-inclusion safe storage/maria/ma_create.c: during recovery, don't log records. Comments. Moving the creation of the first bitmap page to ma_bitmap.c storage/maria/ma_delete_table.c: during recovery, don't log records. Log the end-zero of the dropped table's name, so that recovery can use the string in place without extending it to fit an end zero. storage/maria/ma_loghandler.c: * inwrite_rec_hook also needs access to the MARIA_SHARE, like prewrite_rec_hook. This will be needed to update share->records_diff (in the upcoming patch "recovery of the state"). * LOG_DESC::record_ends_group changed to an enum. * LOG_DESC for LOGREC_REDO_PURGE_BLOCKS and LOGREC_UNDO_ROW_PURGE corrected * Sanja please see the @todo LOG BUG * avoiding DBUG_RETURN(func()) as it gives confusing debug traces. storage/maria/ma_loghandler.h: - log write hooks called while the log's lock is held (inwrite_rec_hook) now need the MARIA_SHARE, like prewrite_rec_hook already had - instead of a bool saying if this record's type ends groups or not, we refine: it may not end a group, it may end a group, or it may be a group in itself. Imagine that we had a physical write failure to a table before we log the UNDO, we still end up in external_lock(F_UNLCK) and then we log a COMMIT: we don't want to consider this COMMIT as ending the group of REDOs (don't want to execute those REDOs during Recovery), that's why we say "COMMIT is a group in itself, it aborts any previous group". This also gives one more sanity check in maria_read_log. storage/maria/ma_recovery.c: New Recovery code, replacing the old pseudocode. Most of maria_read_log moved here. Call-able from ha_maria, but not enabled yet. Compared to the previous version of maria_read_log, some bugs have been fixed, debugging output can go to stdout or a disk file (for now it's useful for me, later it can be changed), execution of REDO_DROP_TABLE, REDO_DELETE_ALL, REDO_PURGE_BLOCKS has been added. Duplicate code has been factored into functions. We abort an unfinished group of records if we see a record which is a group in itself (like COMMIT). No need for maria_panic() after a bug (which caused tables to not be closed) was fixed; if there is yet another bug I prefer to see it. When opening a table for Recovery, set data_file_length and key_file_length to their real physical value (these are the easiest state members to restore :). Warn us if the last page was truncated (but Recovery handles it). MARIA_SHARE::state::state::records is now partly recovered (not idempotent, but works if recreating tables from scracth). When applying a REDO to a page, stamp it with the UNDO's LSN (current_group_end_lsn), not with the REDO's LSN; it makes the table more identical to the original table (easier to compare the two tables in the end). Big thing missing: some types of REDOs are not handled, and the UNDO phase does not exist (missing functions to execute UNDOs to actually rollback). So for now tests are only inserting/deleting a few 100 rows, closing the table and seeing if the log is applied ok; it works. UPDATE not handled. storage/maria/ma_recovery.h: new functions: ma_recover() for recovery from inside ha_maria; _ma_apply_log() for maria_read_log (ma_recover() calls _ma_apply_log()). Btw, we need to not use the word "recover" for REPAIR/maria_chk anymore. storage/maria/ma_rename.c: don't write log records during recovery storage/maria/ma_test2.c: - fail if maria_info() or other subtests find some wrong information - new option -g to skip updates. - init the translog before creating the table, so that log applying can work. - in "#if 0" you'll see some fixed bugs (will be removed). storage/maria/ma_test_all.sh: cleanup files. Test log applying. storage/maria/maria_read_log.c: most of the logic moves to ma_recovery.c to be shared between maria_read_log and recovery-from-inside-mysqld. See ma_recovery.c for additional changes made to the moved code. storage/maria/ma_test_recovery: unit test for Recovery. Tests insert and delete, REDO_UPDATE not yet coded. Script is called from ma_test_all. Can run standalone.	2007-07-26 11:56:21 +02:00