Added part of undo of update row
Extended ma_test1 for recovery testing
Some bug fixes
storage/maria/ha_maria.cc:
Ignore 'state.split' in case of block records
storage/maria/ma_bitmap.c:
Added return value for _ma_bitmap_find_place() for how much data we should put on head page
storage/maria/ma_blockrec.c:
Added undo of deleted row.
- Added logging of CLR_END records in write_block_record()
- Split ma_write_init_block_record() to two functions to get better code reuse
- Added _ma_apply_undo_row_delete()
- Added ma_get_length()
Added 'empty' prototype for undo_row_update()
Fixed bug when moving data withing a head/tail page.
Fixed bug when reading a page with bigger LSN but of different type than was expected.
Store undo_lsn first in CLR_END record
Simplified some code by adding local variables.
Changed log format for UNDO_ROW_DELETE to store total length of used blobs
storage/maria/ma_blockrec.h:
Added prototypes for undo code.
storage/maria/ma_pagecache.c:
Allow plain page to change to LSN page (needed in recovery to apply UNDO)
storage/maria/ma_recovery.c:
Added undo handling of UNDO_ROW_DELETE and UNDO_ROW_UPDATE
storage/maria/ma_test1.c:
Extended --test-undo option to allow us to die after insert or after delete.
Fixed bug in printing key values when using -v
storage/maria/maria_def.h:
Moved some variables around to be getter alignment
Added length_buff buffer to be used during undo handling
Small fixes made.
storage/maria/ma_loghandler.c:
Check of transaction log descriptor table consistance added.\
Incorrect record description fixed.
Compiler warning fixed.
storage/maria/ma_loghandler.h:
fixed ident.
storage/maria/unittest/ma_test_loghandler-t.c:
Suppressing of automatic record writing
storage/maria/unittest/ma_test_loghandler_first_lsn-t.c:
Suppressing of automatic record writing
storage/maria/unittest/ma_test_loghandler_max_lsn-t.c:
Suppressing of automatic record writing
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
Suppressing of automatic record writing
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
Suppressing of automatic record writing
storage/maria/unittest/ma_test_loghandler_noflush-t.c:
Suppressing of automatic record writing
storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
Suppressing of automatic record writing
storage/maria/unittest/ma_test_loghandler_purge-t.c:
Suppressing of automatic record writing
storage/maria/ma_blockrec.c:
Added UNDO handling of insert during recovery
To do this, I also had to add write locking of tail pages during undo phase (As we need to access the same page twice if extents are split over two pages)
Another way to handle the undo of insert would be to store the extent information as part of the UNDO_INSERT block.
storage/maria/ma_blockrec.h:
Added new prototype
storage/maria/ma_loghandler.c:
Changed type of CLR_END (to avoid crash in log handler)
Removed not used variable
storage/maria/ma_loghandler.h:
Added TRN argument to record_execute_in_undo_phase()
storage/maria/ma_pagecache.c:
Hack for undo phase of recovery. During REDO we work with PLAIN pages, but UNDO works with LSN pages, which caused an abort when trying to access a cached page.
storage/maria/ma_recovery.c:
Added execution of UNDO_ROW_INSERT
storage/maria/ma_test1.c:
Added option --test-undo for testing recovery with undo
storage/maria/maria_read_log.c:
Added processing of undos
Renamed HA_EXTRA_PREAPRE_FOR_DELETE to HA_EXTRA_PERPARE_FOR_DROP
Added HA_EXTRA_PREPARE_FOR_RENAME (as we in the code before used HA_EXTRA_PREPARE_FOR_DELETE also for renames which confused things)
Allow multiple write locks for same page by same file handle
Don't write table state if table is not changed
include/my_base.h:
Renamed HA_EXTRA_PREAPRE_FOR_DELETE to HA_EXTRA_PERPARE_FOR_DROP
Added HA_EXTRA_PREPARE_FOR_RENAME (as we in the code before used HA_EXTRA_PREPARE_FOR_DELETE also for renames which confused things)
mysql-test/r/maria.result:
More tests of things that failed in other tests
mysql-test/t/maria.test:
More tests of things that failed in other tests
sql/ha_partition.cc:
HA_EXTRA_PREPARE_FOR_DELETE -> HA_EXTRA_PREPARE_FOR_DROP
Use HA_EXTRA_PREPARE_FOR_RENAME for renames
sql/ha_partition.h:
HA_EXTRA_PREPARE_FOR_DELETE -> HA_EXTRA_PREPARE_FOR_DROP
Use HA_EXTRA_PREPARE_FOR_RENAME for renames
sql/lock.cc:
Fixed comment
sql/sql_table.cc:
Fixed wrong usage of HA_EXTRA_PREAPRE_FOR_DELETE
storage/maria/ha_maria.cc:
Added missing _ma_renable_logging_for_table() (When using with ALTER TABLE + repair index)
Enabled fast generation of index
storage/maria/ma_bitmap.c:
Fixed bug when resetting full pages when page was a tail page
storage/maria/ma_blockrec.c:
Fixed several bugs found by running *.test with maria engine:
During update we keep old changed pages locked with a write lock to be able to reuse them.
- Fixed bug with allocated but not used tail part
- Fixed bug with blob that only had tail part
- Fixed bug when update reused a page (needed multiple write locks for same page)
- Fixed bug when first extent was a tail block
storage/maria/ma_check.c:
Better error message when bitmap is destroyed
storage/maria/ma_close.c:
Only write status if file was changed.
Fixed bug when maria_chk -e file_name changed the file.
storage/maria/ma_dynrec.c:
Removed not used argument to _ma_state_info_read_dsk
storage/maria/ma_extra.c:
HA_EXTRA_PREPARE_FOR_DELETE -> HA_EXTRA_PREPARE_FOR_DROP
Use HA_EXTRA_PREPARE_FOR_RENAME for renames
Only ignore flushing of pages for DROP (not rename)
storage/maria/ma_locking.c:
Removed not used argument to _ma_state_info_read_dsk
storage/maria/ma_open.c:
Removed not used argument to _ma_state_info_read_dsk
storage/maria/ma_pagecache.c:
Allow multiple write locks for same page by same file handle
(Not yet complete, Sanja will fix)
storage/maria/ma_recovery.c:
HA_EXTRA_PREPARE_FOR_DELETE -> HA_EXTRA_PREPARE_FOR_DROP
storage/maria/maria_def.h:
Removed not used argument to _ma_state_info_read_dsk
storage/myisam/mi_extra.c:
HA_EXTRA_PREPARE_FOR_DELETE -> HA_EXTRA_PREPARE_FOR_DROP
Use HA_EXTRA_PREPARE_FOR_RENAME for renames
Only ignore flushing of pages for DROP (not rename)
storage/myisammrg/ha_myisammrg.cc:
HA_EXTRA_PREPARE_FOR_DELETE -> HA_EXTRA_PREPARE_FOR_DROP
Use HA_EXTRA_PREPARE_FOR_RENAME for renames
into desktop.sanja.is.com.ua:/home/bell/mysql/bk/work-maria-purge
storage/maria/ma_loghandler.c:
Auto merged
storage/maria/ma_loghandler.h:
Auto merged
storage/maria/ma_blockrec.c:
Generalized the way update and redo extends the size of a directory record.
This will (for now) ensure that data files are idenitical after normal run and after a apply-log run.
storage/maria/ma_open.c:
Disabled reservation of transid on rows (for now) as these are not yet used.
(I had to disable this as otherwise update thougth rows had grown in size when they hadn't and we had thus different row sizes on update and redo, which caused different block information)
storage/maria/ma_test1.c:
Added comment
storage/maria/ma_test2.c:
Do commit on error/abort
storage/maria/ma_test_all.sh:
Some more testing (to cover a bug that was not found in previous runs)
storage/maria/ma_test_recovery:
More tests
manual merge of ma_recovery.c (too big conflict to resolve in fmtool);
the merged Monty's code allows correct replaying of REDO_PURGE_BLOCKS
and was originally in
monty@mysql.com/narttu.mysql.fi|ChangeSet|20070829060310|44058
storage/maria/ma_recovery.c:
* manually merging Monty's and Sanja's changes of the two last weeks
to my massively modified version of this file. The merged Monty's
code allows correct replaying of REDO_PURGE_BLOCKS and was originally
in monty@mysql.com/narttu.mysql.fi|ChangeSet|20070829060310|44058 .
* Setting the state to "STATE_CHANGED|etc" in Recovery is more
logically done when we update the state in memory (for example
records++).
into gbichot4.local:/home/mysql_src/mysql-maria-for-undo-phase
storage/maria/ha_maria.cc:
Auto merged
storage/maria/ma_blockrec.c:
Auto merged
storage/maria/ma_loghandler.c:
Auto merged
storage/maria/ma_loghandler.h:
Auto merged
storage/maria/ma_loghandler_lsn.h:
Auto merged
storage/maria/maria_chk.c:
Auto merged
storage/maria/maria_read_log.c:
Auto merged
storage/maria/ma_commit.c:
theoretically unneeded, and could cause problems (when trnman_commit_trn()
ends the TRN may have been recycled and be in use by another thread
already, we cannot touch it).
storage/maria/maria_def.h:
just include the existing file
* create page cache before initializing engine and not after, because
Maria's recovery needs a page cache
* make the creation of a bitmap page more crash-resistent
* bugfix (see ma_blockrec.c)
* back to old way: create an 8k bitmap page when creating table
* preparations for the UNDO phase: recreate TRNs
* preparations for Checkpoint: list of dirty pages, testing
of rec_lsn to know if page should be skipped during Recovery
(unused in this patch as no Checkpoint module pushed yet)
* maria_chk tags repaired table with a special LSN
* reworking all around in ma_recovery.c (less duplication)
mysys/my_realloc.c:
noted an issue in my_realloc()
sql/mysqld.cc:
page cache needs to be created before engines are initialized,
because Maria's initialization may do a recovery which needs
the page cache.
storage/maria/ha_maria.cc:
update to new prototype
storage/maria/ma_bitmap.c:
when creating the first bitmap page we used chsize to 8192 bytes then
pwrite (overwrite) the last 2 bytes (8191-8192). If crash between
the two operations, this leaves a bitmap page full without its end
marker. A later recovery may try to read this page and find it
exists and misses a marker and conclude it's corrupted and fail.
Changing the chsize to only 8190 bytes: recovery will then find
the page is too short and recreate it entirely.
storage/maria/ma_blockrec.c:
Fix for a bug: when executing a REDO, if the data page is created,
data_file_length was increased before _ma_bitmap_set():
_ma_bitmap_set() called _ma_read_bitmap_page() which, due to the
increased data_file_length, expected to find a bitmap page on disk
with a correct end marker; if the bitmap page didn't exist already
in fact, this failed. Fixed by increasing data_file_length only after
_ma_read_bitmap_page() has created the new bitmap page correctly.
This bug could happen every time a REDO is about creating a new
bitmap page.
storage/maria/ma_check.c:
empty data file has a bitmap page
storage/maria/ma_control_file.c:
useless parameter to ma_control_file_create_or_open(), just
test if this is recovery.
storage/maria/ma_control_file.h:
new prototype
storage/maria/ma_create.c:
Back to how it was before: maria_create() creates an 8k bitmap page.
Thus (bugfix) data_file_length needs to reflect this instead of being 0.
storage/maria/ma_loghandler.c:
as ma_test1 and ma_test2 now use real transactions and not
dummy_transaction_object, REDO for INSERT/UPDATE/DELETE are always
about real transactions, can assert this.
A function for Recovery to assign a short id to a table.
storage/maria/ma_loghandler.h:
new function
storage/maria/ma_loghandler_lsn.h:
maria_chk tags repaired tables with this LSN
storage/maria/ma_open.c:
* enforce that DMLs on transactional tables use real transactions
and not dummy_transaction_object.
* test if table was repaired with maria_chk (which has to been
seen as an import of an external table into the server), test
validity of create_rename_lsn (header corruption detection)
* comments.
storage/maria/ma_recovery.c:
* preparations for the UNDO phase: recreate TRNs
* preparations for Checkpoint: list of dirty pages, testing
of rec_lsn to know if page should be skipped during Recovery
(unused in this patch as no Checkpoint module pushed yet)
* reworking all around (less duplication)
storage/maria/ma_recovery.h:
a parameter to say if the UNDO phase should be skipped
storage/maria/maria_chk.c:
tag repaired tables with a special LSN
storage/maria/maria_read_log.c:
* update to new prototype
* no UNDO phase in maria_read_log for now
storage/maria/trnman.c:
* a function for Recovery to create a transaction (TRN), needed
in the UNDO phase
* a function for Recovery to grab an existing transaction, needed
in the UNDO phase (rollback all existing transactions)
storage/maria/trnman_public.h:
new functions
Now ma_test1 -M -T and ma_test2 -M -T produces readable, applyable logs
Note: The .MAD file is not binary identical after applying redo compare to a an original file.
(This is becasue we don't have full information which function called PURGE_REDO_BLOCKS).
To verify if a file was correctly applied, we now instead compare row checksums
BitKeeper/etc/ignore:
added storage/maria/tmp/*
include/maria.h:
Added maria_commit() and maria_begin() to be used with external tests
storage/maria/ha_maria.cc:
Ensure maria_def. is read in C mode
storage/maria/ma_blockrec.c:
Fixed redo handling.
_ma_apply_redo_purge_blocks() updated to handle any number of purged blocks
Removed code to make data file idenitcal after redo (can't easily be done). See changeset comments
Now ma_test1 -M -T and ma_test2 -M -T produces readable, applyable logs
storage/maria/ma_commit.c:
More DBUG statements
Moved variable declaration to start of function (portability fix)
Added helper functions 'maria_commit()' and 'maria_begin()'
storage/maria/ma_loghandler.c:
Fixed wrong REDO_PURGE_BLOCKS initialization
storage/maria/ma_recovery.c:
Added UNDO_ROW_UPDATE
Removed wrong setting of lsn (there was no lsn at the used position)
Fixed REDO_PURGE_BLOCKS to handle any number of blocks
storage/maria/ma_test1.c:
Added transaction support (via maria_begin() & maria_commit()) to get a log that can be applied with maria_read_log
storage/maria/ma_test2.c:
Added transaction support (via maria_begin() & maria_commit()) to get a log that can be applied with maria_read_log
storage/maria/ma_test_recovery:
Create temporary files in maria/tmp
Verify files with checksums instead of byte comparisons
storage/maria/maria_chk.c:
When using with -dss we only get filename, records and checksum.
This is useful to do a quick comparision if a files is identical to another one.
storage/maria/maria_def.h:
Added ma_commit()
storage/maria/maria_read_log.c:
Added --help
File number types fixed.
storage/maria/unittest/Makefile.am:
Test of purger added.
storage/maria/unittest/ma_test_loghandler_purge-t.c:
New BitKeeper file ``storage/maria/unittest/ma_test_loghandler_purge-t.c''
Fixed wrong hash function prototype (causes failure on 64 bit systems)
mysql-test/r/rpl_events.result:
Removed wrong merge (result file is now identical as in 5.1 tree)
mysys/lf_hash.c:
Fixed compiler warning
mysys/my_safehash.c:
Fixed wrong hash function prototype (causes failure on 64 bit systems)
storage/maria/unittest/ma_test_loghandler_max_lsn-t.c:
Fixed compiler warning
Fixed bug when doing rnd_read followed by update.
Don't duplicate error messages in log
Initialize transaction object properly. Fixed failure in event*tests when running with maria
mysql-test/mysql-test-run.pl:
Removed warning when running with --external
mysql-test/r/maria.result:
Added back disabled test
Verified that the result changes are correct
mysql-test/t/maria.test:
Added back disabled test
sql/handler.cc:
More debugging.
Simple style change
sql/sql_class.cc:
Initialize transaction object properly. Fixed failure in event*tests when running with maria
storage/maria/ha_maria.cc:
More dbug info
storage/maria/ma_blockrec.c:
Removed not needed line
storage/maria/ma_rrnd.c:
Removed not used code
Ensure that cur_row.lastpos is always set when reading record with rnd. (Fixes failure in maria.test)
storage/maria/maria_def.h:
Don't call maria_print_error() except with EXTRA_DEBUG
(Removes duplicate error messages when somethings goes wrong)
in the file to the file header.
storage/maria/ma_loghandler.h:
Getting maximum LSN of the record which parts written
in the file to the file header.
storage/maria/unittest/Makefile.am:
Test suite for getting max LSN added.
storage/maria/unittest/ma_test_loghandler_first_lsn-t.c:
Spelling fixed.
Cleanup fixed.
storage/maria/unittest/ma_test_loghandler_noflush-t.c:
Cleanup fixed.
storage/maria/unittest/ma_test_loghandler_max_lsn-t.c:
New BitKeeper file ``storage/maria/unittest/ma_test_loghandler_max_lsn-t.c''
storage/maria/ma_checkpoint.c:
Definitions of LSN should be collected in
the one file (ma_loghandler_lsn.h)
storage/maria/ma_loghandler.c:
New calls to get first theoretical and first real LSN.
storage/maria/ma_loghandler.h:
New calls to get first theoretical and first real LSN.
storage/maria/ma_loghandler_lsn.h:
Defined yet another impossible LSN to indicate error.
storage/maria/ma_recovery.c:
The first LSN call changed.
storage/maria/maria_read_log.c:
The first LSN call changed.
storage/maria/unittest/Makefile.am:
New unittest added.
storage/maria/unittest/ma_test_loghandler_first_lsn-t.c:
New BitKeeper file ``storage/maria/unittest/ma_test_loghandler_first_lsn-t.c''
into desktop.sanja.is.com.ua:/home/bell/mysql/bk/work-maria
storage/maria/ma_pagecache.c:
Auto merged
storage/maria/unittest/ma_test_loghandler-t.c:
Auto merged
storage/maria/ma_loghandler.c:
merge
storage/maria/ma_pagecache.h:
merge
Already written pages injection to the cache fixed.
storage/maria/ma_loghandler.c:
Ability to read unflushed data added.
storage/maria/ma_page.c:
Parameters added
storage/maria/ma_pagecache.c:
Already written pages injection to the cache fixed.
Validator for case of page content injection added.
storage/maria/ma_pagecache.h:
Validator for case of page content injection added.
storage/maria/unittest/Makefile.am:
Test of reading unflushed data
storage/maria/unittest/ma_test_loghandler-t.c:
Define fixed.
Restart of the log removed.
storage/maria/unittest/ma_test_loghandler_noflush-t.c:
New BitKeeper file ``storage/maria/unittest/ma_test_loghandler_noflush-t.c''
with engines which support that
* temporarily adding option --global-subst to mysqltest so that
the full testsuite can be run using Maria tables without failing
on trivial differences (like diff in the engine clause of
SHOW CREATE TABLE)
* using recognizable tags for todos of the Maria team
client/mysqltest.c:
temporarily adding option --global-subst: its argument is X,Y.
It replaces all occurrences of X by Y into mysqltest's result
before the comparison with the expected result is done.
This serves for when a test is run with --default-storage-engine=X
where X is not MyISAM: tests using SHOW CREATE TABLE will always fail
because SHOW CREATE TABLE prints X instead of MyISAM. With
--global-subst=X,MyISAM , such trivial differences are eliminated and
test may be reported as passing.
For example, --global-subst=MARIA,MyISAM
This is not good enough for merging into main trees! just for running
many tests and finding bugs now!
mysql-test/mysql-test-run.pl:
new option --mysqltest to pass options to mysqltest (like we have
--mysqld). Used for example like this:
./mtr --mysqltest=--global-subst=MARIA,MyISAM
mysql-test/r/merge.result:
update
mysql-test/t/delayed.test:
run test only with engines which support INSERT DELAYED
mysql-test/t/merge.test:
run test only with MyISAM tables, as they are required by MERGE
sql/sql_delete.cc:
recognizable tag
sql/table.cc:
recognizable tag
storage/maria/ha_maria.cc:
recognizable tag
storage/maria/ma_check.c:
recognizable tag
storage/maria/ma_create.c:
recognizable tag
a table with TRANSACTIONAL=x needs to still have it after TRUNCATE.
No testcase, but without this fix, the frm and the Maria table
got "out of sync" in this case:
create table t1 (a int) row_format=page transactional=0;
truncate table t1;
After TRUNCATE, the Maria table (not the frm) was transactional
(thus logging records, which is wrong).
* fix for non-closed file at end of "maria_chk -r"
sql/table.cc:
"transactionality" needs to be preserved when truncating.
It's behind a if() to not cancel the hack added to mysql_truncate()
today for temporary Maria tables.
storage/maria/ha_maria.cc:
question for Monty (he also has a big mail from me on the same subject)
storage/maria/ma_check.c:
question for Monty (likely bugs)
storage/maria/ma_create.c:
debugging info
storage/maria/ma_open.c:
fix for datafile left open at end of "maria_chk -r":
ma_open_datafile() happens after _ma_bitmap_init(), it sets dfile.file
so needs to set share->bitmap.file.file too (they are copies of
each other). Otherwise it breaks how closing of files works in
BLOCK_RECORD (which is that info.dfile.file is not closed
but share->bitmap.file.file is closed): not setting share->bitmap.file.file
can lead to forgetting to close a file or closing a wrong file.
./mtr --mysqld=--default-storage-engine=maria --mem ps:
I got "can't sync on file UNOPENED" among the messages of REPAIR TABLE;
due to a missing setting of bitmap.file.file to -1.
Maria had two names "Maria" and "MARIA", using one now: "MARIA".
storage/maria/ha_maria.cc:
plug.in uses "MARIA". Some code apparently picks the name from
plug.in (SHOW CREATE TABLE, run ps.test on Maria tables), other from
mysql_declare_plugin (SHOW CREATE TABLE on partitioned tables, run
partition.test with Maria tables), better make names identical.
storage/maria/ma_check.c:
running ps.test on Maria tables I got "can't sync on file UNOPENED"
among the messages of REPAIR TABLE. That was due to maria_repair()
closing the data file, setting info->dfile.file to -1 to prevent
a wrong double close, but forgetting to also set
info->s->bitmap.file.file to -1; it left it unchanged and so,
when close_thread_tables() closed the old version of the repaired
table, _ma_once_end_block_record() tried to fsync the closed
descriptor, resulting in a message.
Basically, when setting info->dfile.file to something it's always
safe and recommended to set bitmap.file.file to the same value
as it's just a copy of the same descriptor see _ma_bitmap_init().
Using change_data_file_descriptor() for that.
Changing that function to use MY_WME as it looks safe.
storage/maria/ma_close.c:
no need to make the index file durable if table is not transactional
of bugfix made to maria_repair() yesterday.
Fail "bk delta" (and thus "bk citool") if an added or modified line
of a C/C++ file has white space at end of line
BitKeeper/triggers/pre-delta:
detection gave false alarm on added newline
storage/maria/ma_check.c:
propagation to maria_repair_by_sort() and maria_repair_parallel()
of bugfix made to maria_repair() yesterday. No effect now as
those two repair variants are never used with BLOCK_RECORD.
"./mtr --mysqld=--default-storage-engine=maria mysqldump".
First problem was use of INSERT DELAYED and MERGE tables without
specifying that the tables to create should always be MyISAM.
After fixing this, no rows were returned by the final SELECT of the
"BUG 19025" portion of the test. Simplified problem was:
LOCK TABLES `t1` WRITE;
/*!40000 ALTER TABLE `t1` DISABLE KEYS */;
INSERT INTO `t1` VALUES ('bla',1000),('bla',1001),('bla',1002);
/*!40000 ALTER TABLE `t1` ENABLE KEYS */;
UNLOCK TABLES;
select * from t1;
The SELECT would find no rows. Reason: ENABLE KEYS does a maria_repair();
but data pages are still in the page cache and not on disk (because
they were not flushed because maria_lock_database(F_UNLCK) was
not called at the end of INSERT because under LOCK TABLES).
At start of maria_repair(), sort_info.filelength is set to the
physical size of the data file (=> too small because pages are in
cache and not on disk).
Then in sort_get_next_record(), when seeing end-of-file, this is done:
sort_param->max_pos= sort_info->filelength;
Further in maria_repair(), this is done:
info->state->data_file_length= sort_param.max_pos;
and so data_file_length is smaller (0) than reality (16384).
This makes SELECT think EOF is where it is not, and thus find
no rows.
This is fixed by flushing all data pages at the start of maria_repair()
(no performance problem is introduced as in common cases where
ALTER TABLE is not under LOCK TABLES, the previous statement did
this flush anyway).
Another reason to do this flush is that, if not doing it, old cached
pages might go down onto the repaired data file at a later point
and thus corrupt it (assume a REPAIR non-QUICK).
A similar bug is fixed:
LOCK TABLES WRITE; INSERT; CHECK TABLE;
reports "Size of datafile is: 0 Should be: 16384"
again because the physical size was read without a preliminary
page cache flush.
mysql-test/r/maria.result:
result update
mysql-test/r/mysqldump.result:
result update
mysql-test/t/maria.test:
adding test for fixed bug in LOCK TABLES + CHECK TABLE + block format.
Disabling portion which hits "incorrect key file" but still
letting it make the test fail (Monty to fix).
mysql-test/t/mysqldump.test:
in places where test expects engine to support INSERT DELAYED and
be includable in a MERGE table, i.e. be MyISAM, we explicitely
ask for MyISAM.
storage/maria/ma_check.c:
Before reading the data file's physical size with my_seek(MY_SEEK_END)
during maria_chk_size() and maria_repair(), we must flush this
data file, otherwise physical size is misleading and leads to
- CHECK TABLE finding the table corrupted ("size of datafile should be"
error)
- ALTER TABLE ENABLE KEYS losing rows (maria_repair()
setting data_file_length to a too small value => later SELECT does
not find rows though they are in the data file).
This fixes the "mysqldump.test" failure.
sort_info.filelength contains the physical size, re-using it.
number 1: "./mtr --mysqld=--default-storage-engine=maria backup"
restored no rows (forgot to flush data pages before my_copy(),
and also the maria_repair() used by ha_maria::restore() needed
a correct data_file_length to not miss rows). [note that BACKUP
TABLE will be removed anyway in 5.2]
number 2: "./mtr --mysqld=--default-storage-engine=maria bootstrap"
caused segfault (uninitialized variable)
number 3: "./mtr --mysqld=--default-storage-engine=maria check"
showed warning in CHECK TABLE (maria_create() created a non-empty
data file with data_file_length==0).
storage/maria/ha_maria.cc:
in ha_maria::backup, need to flush the data file before copying it,
otherwise data misses from the copy (bug 1)
storage/maria/ma_bitmap.c:
when allocating data at the end of the bitmap, best_data is at "end",
should not be left to 0 (bug 2)
storage/maria/ma_check.c:
_ma_scan_block_record() is used in QUICK repair. It relies on
data_file_length. RESTORE TABLE mixes the MAI of an empty table
(so, data_file_length==0) with an non-empty MAD, and does a
QUICK repair; that got fooled (thought it had hit EOF immediately,
so found no records) (bug 1)
storage/maria/ma_create.c:
At the end of maria_create() we have, in the index file,
data_file_length==0, while the data file has a bitmap page (8192).
This inconsistency makes CHECK TABLE rightly complain.
Fixed by not creating a first bitmap page during maria_create()
(also saves disk space) (bug 3) Question for Monty.
storage/maria/ma_extra.c:
A function to flush the data and index files before one can
use OS syscalls (reads, writes) on those files. For example,
ha_maria::backup() does a my_copy() of the data file and so
all cached pieces of this file must be sent to the OS (bug 1)
This function will have to be used elsewhere in Maria, several places
have not been updated when we added pagecache-ing of the data file
(they still only flush the index file), they are probable bugs.
storage/maria/maria_def.h:
new function. Needs to be visible from ha_maria::backup.
storage/maria/ma_range.c:
old code, which is wrong now (key_len is now a local variable,
not initialized). Fixes some problems in running ma_test2.
storage/maria/ma_test2.c:
keypart_map should be used instead of length (fix of bad merge
probably)
storage/maria/ma_test_recovery:
less duplication in this script (one loop instead).
don't auto-log LOGREC_LONG_TRANSACTION_ID in log handler's unit tests,
as they read their log and expect to find only the records they asked
for, and so not LOGREC_LONG_TRANSACTION_ID. All log handler's unit
tests pass. A way to run them faster: when their LONG_BUFFER_SIZE is
too big (1GB) I divide it by ten and then I can run them on /dev/shm
- they are then "instant".
By the way, pushbuild was not displaying anymore "unit: failed" in
the list of all pushes, which participates to the reasons why I didn't
notice the breakage earlier. The other reason being that I was too lazy
to run log handler unit tests on my machine as they took long (hadn't
yet thought about the /dev/shm idea) and so I relied on pushbuild;
Danny has now quickly fixed pushbuild - thanks :)
storage/maria/unittest/ma_test_loghandler-t.c:
don't auto-log LOGREC_LONG_TRANSACTION_ID in log handler's unit tests,
as they read their log and expect to find only the records they asked
for
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
don't auto-log LOGREC_LONG_TRANSACTION_ID in log handler's unit tests,
as they read their log and expect to find only the records they asked
for
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
don't auto-log LOGREC_LONG_TRANSACTION_ID in log handler's unit tests,
as they read their log and expect to find only the records they asked
for
storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
don't auto-log LOGREC_LONG_TRANSACTION_ID in log handler's unit tests,
as they read their log and expect to find only the records they asked
for
Preparation: for the two-checkpoint rule we will flush all pages
which have block->rec_lsn <= last_checkpoint_lsn. Pages with
rec_lsn not yet set (i.e. ==0) were dirtied very recently and so should
not be flushed (otherwise it's not efficient: we should try to let
a page be modified several times in memory before we flush it to disk).
To make this easy, "block->rec_lsn not yet set" is now expressed
with block->rec_lsn==LSN_MAX, not block->rec_lsn==0 anymore.
It is easier this way because LSN_MAX>last_checkpoint_lsn whereas
0<=last_checkpoint_lsn.
storage/maria/ma_blockrec.c:
typo
storage/maria/ma_loghandler.c:
typo
storage/maria/ma_loghandler_lsn.h:
LSN_MAX
storage/maria/ma_pagecache.c:
"block->rec_lsn not yet set" is now expressed by block->rec_lsn==LSN_MAX,
not block_rec_lsn==0.
preparation to enable recoverability of the table's state, more exactly
info->state->checksum. This will require that info->state->checksum
be updated in a inwrite_hook when writing an UNDO record, thus
info->cur_row.checksum needs to be accessible to inwrite_hook, so we
make translog_write_record() accept a MARIA_HA* (info) instead of
MARIA_SHARE* (info->s); with this, we will be able to access
info->cur_row.checksum. Old code which needed the MARIA_SHARE
can derive it from MARIA_HA. Fix for typos and compiler warnings.
storage/maria/ma_blockrec.c:
fix for new loghandler API. Removing strange lines (how could gcc
accept that?)
storage/maria/ma_check.c:
fix for new loghandler API
storage/maria/ma_delete_all.c:
fix for new loghandler API
storage/maria/ma_loghandler.c:
functions now take a MARIA_HA in argument, this is more powerful
than a MARIA_SHARE (MARIA_SHARE can be derived from MARIA_HA, not
the other way around). MARIA_HA will be needed to allow recoverability
of the table's state.
Fixing wrong DBUG_PRINT ('i' is not the id).
When writing the LOGREC_FILE_ID, we don't have a MARIA_HA around,
so we cannot ask translog_write_record() to store the id for us;
we thus store the file's id by ourselves. Alternative would have been
to pass MARIA_HA to translog_assign_id_to_share() but I didn't like it.
storage/maria/ma_loghandler.h:
new loghandler API
storage/maria/tablockman.c:
fix for compiler warning (intptr is int on my machine)
errors when reading a log record which has a 0-length header
(like LOGREC_REDO_DROP_TABLE).
storage/maria/ma_loghandler.c:
Functions reading record's header now don't use 0 to indicate error,
as some valid records have a 0-length header (like REDO_DROP_TABLE).
Instead, negative values are used for EOF and error.
storage/maria/ma_loghandler.h:
functions to read record's header now return an int
(either the length of this header (>=0) or some negative values
for EOF or error).
storage/maria/ma_recovery.c:
update to the new log handler's behaviour. Note the @todo.
storage/maria/maria_read_log.c:
inform when program failed
storage/maria/unittest/ma_test_loghandler-t.c:
update to new log handler's API
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
update to new log handler's API
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
update to new log handler's API
in the pagecache: fixing wrong assertion and a test case.
maria.test and ps_maria.test still fail.
mysql-test/r/maria.result:
result update
mysql-test/t/maria.test:
added test for incorrect assert on page's type
storage/maria/ma_pagecache.c:
It is allowed to change unknown type to any legal type of page in the
pagecache.
maria.test and ps_maria.test still fail.
mysys/mf_keycache.c:
split string annoys some compilers
storage/maria/ha_maria.cc:
fix for compiler warnings
storage/maria/ma_test1.c:
porting Serg's fix for BUG#30094 to Maria
storage/maria/ma_test2.c:
porting Serg's fix for BUG#30094 to Maria
storage/maria/ma_test3.c:
porting Serg's fix for BUG#30094 to Maria
storage/maria/ma_test_recovery:
don't print ma_test1's messages if no problem
Jul 7th). "maria.test" and "ps_maria.test" still fail;
"ma_test_all" starts failing (MyISAM has the same issue see BUG#30094).
include/maria.h:
merging MyISAM changes into Maria
mysys/mf_keycache.c:
mi_test_all showed "floating point exception", this was already
fixed in the latest 5.1, importing fix.
sql/item_xmlfunc.cc:
compiler warning (already fixed in latest 5.1)
storage/maria/ha_maria.cc:
merging MyISAM changes into Maria. See #ifdef ASK_MONTY.
storage/maria/ha_maria.h:
merging MyISAM changes into Maria
storage/maria/ma_cache.c:
merging MyISAM changes into Maria
storage/maria/ma_check.c:
merging MyISAM changes into Maria
storage/maria/ma_create.c:
merging MyISAM changes into Maria
storage/maria/ma_dynrec.c:
merging MyISAM changes into Maria
storage/maria/ma_extra.c:
merging MyISAM changes into Maria
storage/maria/ma_ft_boolean_search.c:
merging MyISAM changes into Maria
storage/maria/ma_ft_nlq_search.c:
merging MyISAM changes into Maria
storage/maria/ma_info.c:
merging MyISAM changes into Maria
storage/maria/ma_key.c:
merging MyISAM changes into Maria
storage/maria/ma_loghandler.c:
compiler warning (part->length is size_t)
storage/maria/ma_open.c:
merging MyISAM changes into Maria
storage/maria/ma_preload.c:
merging MyISAM changes into Maria
storage/maria/ma_range.c:
merging MyISAM changes into Maria
storage/maria/ma_rkey.c:
merging MyISAM changes into Maria
storage/maria/ma_rt_index.c:
merging MyISAM changes into Maria
storage/maria/ma_rt_key.c:
merging MyISAM changes into Maria
storage/maria/ma_rt_split.c:
merging MyISAM changes into Maria
storage/maria/ma_search.c:
merging MyISAM changes into Maria
storage/maria/ma_sort.c:
merging MyISAM changes into Maria
storage/maria/maria_def.h:
merging MyISAM changes into Maria
found in pushbuild on sapsrv1. Some not fixed as not repeatable
on my machine (32/64 bit issue?).
Fixes for some test failures:
- "maria-connect" now passes;
- "maria": after fixing the obvious reasons for failures, the test
went further and hit a more complex issues: difference in the output
of EXPLAIN output; not fixed;
- "ps_maria" still crashes in assertion
mysqld: ha_maria.cc:1627: virtual int ha_maria::index_read(uchar*, const uchar*, uint, ha_rkey_function): Ass
ertion `inited == INDEX' failed, as already observable in pushbuild.
All this might just be due to an incomplete merge of MyISAM changes
into Maria when 5.1 was last merged to mysql-maria.
include/my_global.h:
temporary fix until next merge of 5.1; without this it does not build
mysql-test/r/maria-connect.result:
position changed
mysql-test/t/maria-connect.test:
If one wants to use the binlog it has to ask for it.
1582 is not used for dup entry error anymore (it was in older 5.1).
Size of first event in binlog was increased by 4 (when the new type
of event "gap" was added).
mysql-test/t/maria.test:
1582 not used anymore in this case
storage/maria/ha_maria.cc:
engine now has to say what binlogging it supports
storage/maria/ma_blockrec.c:
fix for compiler warnings ("comparison is always true" or "always
false")
storage/maria/ma_loghandler.c:
fix for compiler warnings (comparing char* to uchar*)
storage/maria/ma_packrec.c:
fix for compiler warning (fix simply merged from MyISAM)
storage/maria/ma_pagecache.c:
info_check_pin() was not used so gave a compiler warning.
storage/maria/ma_pagecache.h:
fixing typo from the last 5.1->maria merge.
storage/maria/ma_recovery.c:
my_free() has a void* argument, so why cast. byte->uchar.
storage/maria/ma_search.c:
fix for compiler warning (fix simply merged from MyISAM)
storage/maria/maria_read_log.c:
gptr->uchar*
storage/maria/trnman.c:
probable fix for warning found in pushbuild (but not on my machine):
storage/maria/trnman.c: 142
passing argument 6 of \u2018lf_hash_init\u2019 from incompatible pointer type
on sapsrv1.
sql/handler.cc:
Auto merged
storage/maria/ha_maria.cc:
Auto merged
storage/maria/ma_blockrec.c:
Auto merged
storage/maria/ma_close.c:
Auto merged
storage/maria/ma_create.c:
Auto merged
storage/maria/ma_loghandler.c:
Auto merged
storage/maria/ma_loghandler.h:
Auto merged
storage/maria/ma_test2.c:
Auto merged
storage/maria/maria_read_log.c:
will merge by hand
Unit test for recovery: runs ma_test1 and ma_test2 (both only with
INSERTs and DELETEs; UPDATEs disabled as not handled by recovery)
then moves the tables elswhere; recreates tables from the log, and
compares and fails if there is a difference. Passes now.
Most of maria_read_log.c moved to ma_recovery.c, as it will be re-used
for recovery-from-ha_maria.
Bugfixes of applying of REDO_INSERT, REDO_PURGE_ROW.
Applying of REDO_PURGE_BLOCKS, REDO_DELETE_ALL, REDO_DROP_TABLE,
UNDO_ROW_INSERT (in REDO phase only, i.e. just doing records++),
UNDO_ROW_DELETE, UNDO_ROW_PURGE.
Code cleanups.
Monty: please look for "QQ". Sanja: please look for "Sanja".
Future tasks: recovery of the bitmap (easy), recovery of the state
(make it idempotent), more REDOs (Monty to work on
REDO_UPDATE?), UNDO phase...
Pushing this cset as it looks safe, contains test and bugfixes which
will help Monty implement applying of REDO_UPDATE.
sql/handler.cc:
typo
storage/maria/Makefile.am:
Adding ma_test_recovery (which ma_test_all invokes, and which can
also be run alone). Most of maria_read_log.c moved to ma_recovery.c
storage/maria/ha_maria.cc:
comments
storage/maria/ma_bitmap.c:
fixing comments. 2 -> sizeof(maria_bitmap_marker).
Bitmap-related part of _ma_initialize_datafile() moves in bitmap module.
Now putting the "bm" signature when creating the first bitmap page
(it used to happen only at next open, but that
caused an annoying difference when testing Recovery if the original
run didn't open the table, and it looks more
logical like this: it goes to disk only with its signature correct);
see the "QQ" comment towards the _ma_initialize_data_file() call
in ma_create.c for more).
When reading a bitmap page, verify its signature (happens when normally
using the table or when CHECKing it; not when REPAIRing it).
storage/maria/ma_blockrec.c:
* no need to sync the data file if table is not transactional
* Comments, code cleanup (log-related data moved to log-related code
block, int5store->page_store).
* Store the table's short id into LOGREC_UNDO_ROW_PURGE, like we
do for other records (though this record will soon be replaced
with a CLR).
* If "page" is 1 it means the page which extends from byte
page*block_size+1 to (page+1)*block_size (byte number 1 being
the first byte of the file). The last byte of the file is
data_file_length (same convention).
A new page needs to be created if the last byte of the page is
beyond the last byte of the file, i.e.
(page+1)*block_size+1 > data_file_length, so we correct the test
(bug found when testing log applying for ma_test1 -M -T --skip-update).
* update the page's LSN when removing a row from it during
execution of a REDO_PURGE_ROW record (bug found when testing log
applying for ma_test1 -M -T --skip-update).
* applying of REDO_PURGE_BLOCKs (limited to a one-page range for now).
storage/maria/ma_blockrec.h:
new functions. maria_bitmap_marker does not need to be exported.
storage/maria/ma_close.c:
we can always flush the table's state when closing the last instance
of the table. And it is needed for maria_read_log (as it does
not use maria_lock_database()).
storage/maria/ma_control_file.c:
when in Recovery, some assertions should not be used.
storage/maria/ma_control_file.h:
double-inclusion safe
storage/maria/ma_create.c:
during recovery, don't log records. Comments.
Moving the creation of the first bitmap page to ma_bitmap.c
storage/maria/ma_delete_table.c:
during recovery, don't log records. Log the end-zero of the dropped
table's name, so that recovery can use the string in place without
extending it to fit an end zero.
storage/maria/ma_loghandler.c:
* inwrite_rec_hook also needs access to the MARIA_SHARE, like
prewrite_rec_hook. This will be needed to update
share->records_diff (in the upcoming patch "recovery of the state").
* LOG_DESC::record_ends_group changed to an enum.
* LOG_DESC for LOGREC_REDO_PURGE_BLOCKS and LOGREC_UNDO_ROW_PURGE
corrected
* Sanja please see the @todo LOG BUG
* avoiding DBUG_RETURN(func()) as it gives confusing debug traces.
storage/maria/ma_loghandler.h:
- log write hooks called while the log's lock is held (inwrite_rec_hook)
now need the MARIA_SHARE, like prewrite_rec_hook already had
- instead of a bool saying if this record's type ends groups or not,
we refine: it may not end a group, it may end a group, or it may
be a group in itself. Imagine that we had a physical write failure
to a table before we log the UNDO, we still end up in
external_lock(F_UNLCK) and then we log a COMMIT: we don't want
to consider this COMMIT as ending the group of REDOs (don't want
to execute those REDOs during Recovery), that's why we say "COMMIT
is a group in itself, it aborts any previous group". This also
gives one more sanity check in maria_read_log.
storage/maria/ma_recovery.c:
New Recovery code, replacing the old pseudocode.
Most of maria_read_log moved here.
Call-able from ha_maria, but not enabled yet.
Compared to the previous version of maria_read_log, some bugs have
been fixed, debugging output can go to stdout or a disk file (for now
it's useful for me, later it can be changed), execution of
REDO_DROP_TABLE, REDO_DELETE_ALL, REDO_PURGE_BLOCKS has been added. Duplicate code
has been factored into functions. We abort an unfinished group
of records if we see a record which is a group in itself (like COMMIT).
No need for maria_panic() after a bug (which caused tables to not
be closed) was fixed; if there is yet another bug I prefer to see it.
When opening a table for Recovery, set data_file_length
and key_file_length to their real physical value (these are the
easiest state members to restore :). Warn us if the last page
was truncated (but Recovery handles it).
MARIA_SHARE::state::state::records is now partly recovered (not
idempotent, but works if recreating tables from scracth).
When applying a REDO to a page, stamp it with the UNDO's LSN
(current_group_end_lsn), not with the REDO's LSN; it makes
the table more identical to the original table (easier to compare
the two tables in the end).
Big thing missing: some types of REDOs are not handled,
and the UNDO phase does not exist (missing functions to execute UNDOs
to actually rollback). So for now tests are only inserting/deleting
a few 100 rows, closing the table and seeing if the log is applied ok;
it works. UPDATE not handled.
storage/maria/ma_recovery.h:
new functions: ma_recover() for recovery from inside ha_maria;
_ma_apply_log() for maria_read_log (ma_recover() calls _ma_apply_log()).
Btw, we need to not use the word "recover" for REPAIR/maria_chk anymore.
storage/maria/ma_rename.c:
don't write log records during recovery
storage/maria/ma_test2.c:
- fail if maria_info() or other subtests find some wrong information
- new option -g to skip updates.
- init the translog before creating the table, so that log applying
can work.
- in "#if 0" you'll see some fixed bugs (will be removed).
storage/maria/ma_test_all.sh:
cleanup files. Test log applying.
storage/maria/maria_read_log.c:
most of the logic moves to ma_recovery.c to be shared between
maria_read_log and recovery-from-inside-mysqld.
See ma_recovery.c for additional changes made to the moved code.
storage/maria/ma_test_recovery:
unit test for Recovery. Tests insert and delete,
REDO_UPDATE not yet coded.
Script is called from ma_test_all. Can run standalone.
into labbari.dsl.inet.fi:/home/my/bk/mysql-maria.prod
mysys/thr_lock.c:
Auto merged
storage/csv/ha_tina.cc:
Auto merged
storage/csv/ha_tina.h:
Auto merged
storage/maria/ma_blockrec.c:
Auto merged
storage/maria/ma_check.c:
Auto merged
storage/maria/ma_loghandler.c:
Auto merged
storage/maria/trnman.c:
Auto merged
storage/myisam/mi_locking.c:
Auto merged
sql/lock.cc:
SCCS merged
it means, {update,restore}_status() should be called in external_lock,
not in thr_unlock. Only affects storage engines that support
TL_WRITE_CONCURRENT.
mode) so ma_test_all works again; more error detection in ma_test_all;
maria_control renamed to maria_log_control (Monty's suggestion,
so that a "rm maria_log*" removes all log-related files). Disabling
failing wrong assertion.
storage/maria/ma_blockrec.c:
disabling assertion which fails because cur_block is a
local variable not initialized
storage/maria/ma_check.c:
comment
storage/maria/ma_control_file.h:
control file renamed
storage/maria/ma_loghandler.c:
assertions needed only in multi-threaded mode (ma_test1 and ma_test2
are single-threaded, it's ok for them to use dummy_transaction_object
with transactional tables: trn->rec_lsn can be set without interfering
with other threads).
storage/maria/ma_test_all.sh:
got caught by failures in some ma_test1 runs, which I didn't see
because ma_test_all returned 0 and I didn't scroll up in the window;
now using "set -e" to avoid that. Also testing that
we get the errors and warnings we expect.
storage/maria/unittest/Makefile.am:
maria_control renamed
