include/maria.h is a common header included in half of the server,
if should only contain definitions and declarations that are
used outside of storage/maria
internal definitions and declarations should be in maria_def.h
also remove few duplicate declarations
A read-only storage engine that stores it's data in (aws) S3
To store data in S3 one could use ALTER TABLE:
ALTER TABLE table_name ENGINE=S3
libmarias3 integration done by Sergei Golubchik
libmarias3 created by Andrew Hutchings
This commit is based on the work of Michal Schorm, rebased on the
earliest MariaDB version.
Th command line used to generate this diff was:
find ./ -type f \
-exec sed -i -e 's/Foundation, Inc., 59 Temple Place, Suite 330, Boston, /Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, /g' {} \; \
-exec sed -i -e 's/Foundation, Inc. 59 Temple Place.* Suite 330, Boston, /Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, /g' {} \; \
-exec sed -i -e 's/MA.*.....-1307.*USA/MA 02110-1335 USA/g' {} \; \
-exec sed -i -e 's/Foundation, Inc., 59 Temple/Foundation, Inc., 51 Franklin/g' {} \; \
-exec sed -i -e 's/Place, Suite 330, Boston, MA.*02111-1307.*USA/Street, Fifth Floor, Boston, MA 02110-1335 USA/g' {} \; \
-exec sed -i -e 's/MA.*.....-1307/MA 02110-1335/g' {} \;
Fixed wrong calculation of buffer sizes. ulong datatype was used wrongly,
as were the casts to ulong. Buffer sizes should be of type size_t,
not ulong, or bad things happen on 64 bit Windows.
This patch changes pagecache struct to use size_t/ssize_t
where long/ulong were previously used. Also, removed several casts.
Reason for the problem was that the hash of changed files in the key cache was too small (was 128). Fixed by making the hash size larger and changeable.
- Introduced key-cache-file-hash-size (default 512) for MyISAM and aria_pagecache_file_hash_size (default 512) for Aria.
- Added new status variable "Feature_delay_key_write" which counts number of tables opened that are using delay_key_write
mysql-test/r/features.result:
Added test of Feature_delay_key_write
mysql-test/r/key_cache.result:
Updated tests as the number of blocks has changed
mysql-test/r/mysqld--help.result:
Updated result
mysql-test/suite/maria/maria3.result:
Updated result
mysql-test/suite/sys_vars/r/key_cache_file_hash_size_basic.result:
Test new variable
mysql-test/suite/sys_vars/t/aria_pagecache_file_hash_size_basic.test:
Test new variable
mysql-test/suite/sys_vars/t/key_cache_file_hash_size_basic.test:
Test new variable
mysql-test/t/features.test:
Added test of Feature_delay_key_write
mysql-test/t/key_cache.test:
Updated tests as the number of blocks has changed
mysys/mf_keycache.c:
Made CHANGED_BLOCKS_HASH dynamic
sql/handler.cc:
Updated call to init_key_cache()
sql/mysqld.cc:
Added "Feature_delay_key_write"
Added support for key-cache-file-hash-size
sql/mysqld.h:
Added support for key-cache-file-hash-size
sql/sql_class.h:
Added feature_files_opened_with_delayed_keys
sql/sys_vars.cc:
Added key_cache_file_hash_size
storage/maria/ha_maria.cc:
Added pagecache_file_hash_size
Added counting of files with delay_key_write
storage/maria/ma_checkpoint.c:
Fixed compiler warning
storage/maria/ma_pagecache.c:
Made PAGECACHE_CHANGED_BLOCKS_HASH into a variable
storage/maria/ma_pagecache.h:
Made PAGECACHE_CHANGED_BLOCKS_HASH into a variable
storage/maria/ma_rt_test.c:
Updated parameters for init_pagecache()
storage/maria/ma_test1.c:
Updated parameters for init_pagecache()
storage/maria/ma_test2.c:
Updated parameters for init_pagecache()
storage/maria/ma_test3.c:
Updated parameters for init_pagecache()
storage/maria/maria_chk.c:
Updated parameters for init_pagecache()
storage/maria/maria_ftdump.c:
Updated parameters for init_pagecache()
storage/maria/maria_pack.c:
Updated parameters for init_pagecache()
storage/maria/maria_read_log.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_pagecache_consist.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_pagecache_rwconsist.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_pagecache_rwconsist2.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_pagecache_single.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_test_loghandler-t.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_test_loghandler_first_lsn-t.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_test_loghandler_max_lsn-t.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_test_loghandler_noflush-t.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_test_loghandler_nologs-t.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
Updated parameters for init_pagecache()
storage/maria/unittest/ma_test_loghandler_purge-t.c:
Updated parameters for init_pagecache()
storage/myisam/ha_myisam.cc:
Added counting of files with delay_key_write
storage/myisam/mi_check.c:
Updated call to init_key_cache()
storage/myisam/mi_test1.c:
Updated call to init_key_cache()
storage/myisam/mi_test2.c:
Updated call to init_key_cache()
storage/myisam/mi_test3.c:
Updated call to init_key_cache()
storage/myisam/mi_test_all.sh:
Fixed broken test
storage/myisam/myisam_ftdump.c:
Updated call to init_key_cache()
storage/myisam/myisamchk.c:
Updated call to init_key_cache()
storage/myisam/myisamlog.c:
Updated call to init_key_cache()
sql/sql_insert.cc:
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
******
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
sql/sql_table.cc:
small cleanup
******
small cleanup
- Fix for LP#700623 "Aria recovery: ma_blockrec.c:3930: _ma_update_at_original_place: Assertion `block->org_bitmap_value == _ma_bitmap_get_page_bits(info, &info->s->bitmap, page)' failed"
- Issue was that when deleting a tail page where all index entries where full, the page was marked wrongly in the bitmap.
- If debug_assert_if_crashed_table is set, we now crash when we find Aria corrupted.
- Write more information if we find something wrong with the bitmap.
- Fixed that REPAIR also can fix wrong create_rename_lsn issues (a very unlikely event)
- Define STATE_CRASHED_FLAGS as set of all CRASHED flags (to simplify code)
storage/maria/ha_maria.cc:
Mark the normal page cache (not the page cache for the logs) so that we can request extra debugging for it.
Copy the value of debug_assert_if_crashed_table to maria_assert_if_crashed_table so that we can request a crash at exactly the point where we find Aria corrupted.
Use STATE_CRASHED_FLAGS
storage/maria/ma_bitmap.c:
Made bits_to_txt extern so that we can use this in maria_chk
Added extra information to the log files to be able to easier find bitmap failures in recovery. (When compiling with -DEXTRA_DEBUG_BITMAP)
Added _ma_get_bitmap_description() to request a clear text description of the bitmap.
Simplify _ma_check_bitmap_data(), as we know the bitmap pattern in the caller.
storage/maria/ma_blockrec.c:
In delete_head_or_tail(), fixed a bug where we sent wrong information to _ma_bitmap_set() if the directory was full for a page that should be freed.
This fixed LP#700623 (failure in bitmap found during recovery)
storage/maria/ma_blockrec.h:
Added definitions for _ma_get_bitmap_description() and bits_to_txt
storage/maria/ma_check.c:
Simplify call to _ma_check_bitmap_data().
Write more information if we find something wrong with the bitmap.
Moved getting clear text information about the bitmap to ma_bitmap.c::_ma_get_bitmap_description()
storage/maria/ma_checkpoint.c:
More asserts
storage/maria/ma_create.c:
Fix wrong create_rename_lsn during repair.
(Create_rename_lsn can be too big if someone restores an old maria_log_file after an Aria file was created)
storage/maria/ma_delete.c:
Call _ma_set_fatal_error() in case of crashed file
Remove not needed test of save_errno == HA_ERR_KEY_NOT_FOUND. (Handled by other code
storage/maria/ma_extra.c:
Call _ma_set_fatal_error() in case of crashed file
Reset share->bitmap.changed_not_flushed to not cause new ASSERTS to trigger.
Added _ma_file_callback_to_id() for writing share->id to log file in case of DEBUG logging.
storage/maria/ma_init.c:
Destroy also translog if it's readonly (as when called by maria_read_log -d)
storage/maria/ma_key.c:
Call _ma_set_fatal_error() in case of crashed file
storage/maria/ma_key_recover.c:
STATE_CRASHED -> STATE_CRASHED_FLAGS
storage/maria/ma_keycache.c:
Call _ma_set_fatal_error() in case of crashed file
storage/maria/ma_locking.c:
Call _ma_set_fatal_error() in case of crashed file.
Added _ma_set_fatal_error()
storage/maria/ma_open.c:
Call _ma_set_fatal_error() in case of crashed file
storage/maria/ma_page.c:
Call _ma_set_fatal_error() in case of crashed file
storage/maria/ma_pagecache.c:
Added extra information to log file to simply debugging of bitmap errors.
storage/maria/ma_pagecache.h:
Added extra_debug flag to allow marking of row and index cache for extra logging (for debugging).
storage/maria/ma_panic.c:
Flush both data and index blocks in case of HA_PANIC_CLOSE
Fixed wrong position of 'break'. (Not critical for MariaDB as MariaDB never uses this code)
storage/maria/ma_recovery_util.c:
Avoid writing extra not needed \n to DBUG log.
storage/maria/ma_rkey.c:
Call _ma_set_fatal_error() in case of crashed file
storage/maria/ma_search.c:
Call _ma_set_fatal_error() in case of crashed file
storage/maria/ma_static.c:
Define maria_assert_if_crashed_table
storage/maria/ma_update.c:
Call _ma_set_fatal_error() in case of crashed file.
The new code also avoids a problem where we before would print the error twice.
storage/maria/ma_write.c:
Call _ma_set_fatal_error() in case of crashed file
storage/maria/maria_chk.c:
STATE_CRASHED -> STATE_CRASHED_FLAGS
storage/maria/maria_def.h:
Added STATE_CRASHED_PRINTED to avoid giving error message about crash twice.
Added STATE_CRASHED_FLAGS to be able to easily detect and set all CRASHED related flags.
Added prototypes for new functions.
storage/myisam/mi_panic.c:
Fixed wrong position of 'break'. (Not critical for MariaDB as MariaDB never uses this code)
- Don't delete pages without flushing that has had a tail or head information in pagecache_delete()
This fixes a case where REPAIR could find old deleted rows.
storage/maria/ha_maria.cc:
Remove calls to depricated function ha_statistic_increment
storage/maria/ma_blockrec.c:
Don't delete pages without flushing that has had a tail or head information in pagecache_delete()
storage/maria/ma_pagecache.c:
Added possibility to mark pages to not be deleted by pagecache_delete() without beeing flushed.
storage/maria/ma_pagecache.h:
Added new prototype
in write_changed_bitmap(), and page cache forbids that. Here we make the page
cache more relaxed. Original patch by Sanja, simplified by me as limited to
not-locked. See comment of ma_bitmap.c.
With that, maria_stress.yy runs until hitting BUG 39665.
storage/maria/ma_bitmap.c:
A thread which unpins bitmap pages in _ma_bitmap_unpin_all() sometimes
hit an assertion in the page cache (info!=0 in remove_pin()) which states
that you can unpin/unlock only what *you* have pinned/locked.
Fixed by setting the new last parameter of pagecache_unlock_by_link()
to TRUE in _ma_bitmap_unpin_all().
storage/maria/ma_blockrec.c:
new prototype and splitting assertion in three (3rd one fires: BUG 39665)
storage/maria/ma_check.c:
new prototype
storage/maria/ma_key_recover.c:
new prototype
storage/maria/ma_loghandler.c:
new prototype
storage/maria/ma_pagecache.c:
Allow a thread to unpin, with pagecache_unlock_by_link(), a non-locked page pinned by others.
This is a hack for _ma_bitmap_unpin_all() which needs to unpin pages which were
pinned by other threads in write_changed_bitmap().
storage/maria/ma_pagecache.h:
new prototype
storage/maria/ma_preload.c:
new prototype
storage/maria/unittest/ma_pagecache_rwconsist.c:
new prototype
storage/maria/unittest/ma_pagecache_single.c:
new prototype
fixes for ma_test_recovery.pl to work in release builds too:
- bugfix in maria_zerofill_index()
- applying of LOGREC_INSERT_ROW_BLOBS now zeroes unused end of non-full
blob page (a mutation of tail page when it takes >75% of maria_block_size)
like write_full_pages() does.
storage/maria/ma_blockrec.c:
When we write a non-full blob page at run-time, we zero the rest of
it (see write_full_pages()). We now do the same in
_ma_apply_redo_insert_row_blobs(): this is consistent and helps
having log-applying produce the same page as run-time.
storage/maria/ma_check.c:
maria_zerofill_index() was wrong: it didn't zero certain bytes
because it believed that the count of relevant bytes is
_ma_get_page_used(share, buff) + share->keypage_header,
whereas it's only the first term.
storage/maria/ma_pagecache.c:
typo
storage/maria/ma_pagecache.h:
typo
storage/maria/maria_chk.c:
enable --zerofill-keep-lsn in all builds
storage/maria/unittest/ma_test_recovery.pl:
now even release builds can run ma_test_recovery.pl, because zerofill
makes tables of this test (made by run-time and by log-applying)
identical.
Added support for delete by link.
Aded level ("hits") management functions.
storage/maria/ma_pagecache.h:
Added support for delete by link.
Aded level ("hits") management functions.
storage/maria/unittest/ma_pagecache_single.c:
Test of delete by link.
(fix is keeping the real TRN through a disable_logging/reenable cycle)
- fix for pagecache assertion failure in ps/type_ranges with default
engine=maria (fix is in sql_insert.cc)
- when reenabling logging we must either flush all dirty pages,
or at least verify (in debug build) that there are none. For example
a bulk insert with single UNDO_BULK_INSERT must flush them, no matter
if it uses repair or not (bugfix)
- UNDO_BULK_INSERT_WITH_REPAIR is also used with repair, changes name
mysql-test/r/maria.result:
tests for bugs fixed
mysql-test/t/maria.test:
tests for bugs fixed
sql/sql_insert.cc:
Bugfix: even if select_create::prepare() failed to create the 'table' object
we still have to re-enable logging.
storage/maria/ha_maria.cc:
Bugfix: when a transactional table does a bulk insert without
repair, it still sometimes skips logging of REDOs thus needs a full
flush and sync at the end. Not if repair is done, as repair does
it internally already (see end of maria_repair*()).
storage/maria/ha_maria.h:
variable now can have 3 states not 2
storage/maria/ma_bitmap.c:
name change
storage/maria/ma_blockrec.c:
name change
storage/maria/ma_blockrec.h:
name change
storage/maria/ma_check.c:
* When maria_repair() re-enables logging it does not need to ask for
a flush&sync as it did it by itself already a few lines before.
* the log record of bulk insert can be used even without repair
* disable logging in maria_zerofill(): without that, it puts LSN pages
in the cache, so when it flushes them it flushes the log; the change
makes auto-ha_maria::zerofill-if-moved faster (no log flush).
storage/maria/ma_key_recover.c:
name change
storage/maria/ma_loghandler.c:
name change
storage/maria/ma_loghandler.h:
name change
storage/maria/ma_pagecache.c:
A function, to check in debug builds that no dirty pages exist for a file.
storage/maria/ma_pagecache.h:
new function (nothing in non-debug)
storage/maria/ma_recovery.c:
_ma_tmp_disable_logging() sets info->trn to dummy_transaction_object
when needed now. The changes done here about info->trn are to allow
a table to retain its original, real TRN through a disable/reenable
cycle (see replication scenario in _ma_reenable_logging_for_table()).
When we reenable, we offer the caller to flush and sync the table;
if the caller doesn't accept our offer, we verify that it's ok
(no REDOs => no dirty pages are allowed to exist).
storage/maria/maria_chk.c:
comment
storage/maria/maria_def.h:
new names
mysql-test/suite/rpl/r/rpl_stm_maria.result:
result (it used to crash)
mysql-test/suite/rpl/t/rpl_stm_maria.test:
Test of replication-specific Maria bug fixed
Abort mysqld if Maria engine didn't start and we are using Maria for temporary tables
Fixed bug that caused update of big blobs to crash
Use pagecache_page_no_t as type for pages (to get rid of compiler warnings)
Added cast to get rid of compiler warning
Fixed wrong types of variables and arguments that caused lost information
Fixed wrong DBUG_ASSERT() that caused REDO of big blobs to fail
Removed some historical ifdefs that caused problem with windows compilations
BUILD/SETUP.sh:
Added --with-maria-tmp-tables
include/maria.h:
Use pagecache_page_no_t as type for pages
Use my_bool as parameter for 'rep_quick' option
include/my_base.h:
Added comment
mysql-test/r/maria-big.result:
Added test that uses big blobs
mysql-test/t/maria-big.test:
Added test that uses big blobs
sql/mysqld.cc:
Abort mysqld if Maria engine didn't start and we are using Maria for temporary tables
sql/sql_class.h:
Don't use Maria for temporary tables if --with-maria-tmp-tables is not defined
sql/sql_select.cc:
Don't use Maria for temporary tables if --with-maria-tmp-tables is not defined
storage/maria/ha_maria.cc:
Fixed compiler warnings reported by MCC
- Fixed usage of wrong types that caused data loss
- Changed parameter for rep_quick to my_bool
- Added safe casts
Fixed indentation
storage/maria/ma_bitmap.c:
Use pagecache_page_no_t as type for pages
Fixed compiler warnings
Fixed bug that caused update of big blobs to crash
storage/maria/ma_blockrec.c:
Use pagecache_page_no_t as type for pages
Use my_bool as parameter for 'rep_quick' option
Fixed compiler warnings
Fixed wrong DBUG_ASSERT()
storage/maria/ma_blockrec.h:
Use pagecache_page_no_t as type for pages
storage/maria/ma_check.c:
Fixed some wrong parameters where we didn't get all bits for test_flag
Changed rep_quick to be of type my_bool
Use pagecache_page_no_t as type for pages
Added cast's to get rid of compiler warnings
Changed type of record_pos to get rid of compiler warning
storage/maria/ma_create.c:
Added safe cast's to get rid of compiler warnings
storage/maria/ma_dynrec.c:
Fixed usage of wrong type
storage/maria/ma_key.c:
Fixed compiler warning
storage/maria/ma_key_recover.c:
Use pagecache_page_no_t as type for pages
storage/maria/ma_loghandler_lsn.h:
Added cast's to get rid of compiler warnings
storage/maria/ma_page.c:
Changed variable name from 'page' to 'pos' as it was an offset and not a page address
Moved page_size inside block to get rid of compiler warning
storage/maria/ma_pagecache.c:
Fixed compiler warnings
Replaced compile time assert with TODO
storage/maria/ma_pagecache.h:
Use pagecache_page_no_t as type for pages
storage/maria/ma_pagecrc.c:
Allow bitmap pages that is all zero
storage/maria/ma_preload.c:
Added cast to get rid of compiler warning
storage/maria/ma_recovery.c:
Changed types to get rid of compiler warnings
Use bool for quick_repair to get rid of compiler warning
Fixed some variables that was wrongly declared (not enough precission)
Added cast to get rid of compiler warning
storage/maria/ma_test2.c:
Remove historical undefs
storage/maria/maria_chk.c:
Changed rep_quick to bool
Fixed wrong parameter to maria_chk_data_link()
storage/maria/maria_def.h:
Use pagecache_page_no_t as type for pages
storage/maria/maria_pack.c:
Renamed isam -> maria
storage/maria/plug.in:
Added option --with-maria-tmp-tables
storage/maria/trnman.c:
Added cast to get rid of compiler warning
storage/myisam/mi_test2.c:
Remove historical undefs
Fixed failure in unittest/ma_test_loghandler_pagecache-t
Initialize pagecache callbacks explictily, not with pagecache_init().
This is to make things more readable and for the future to make more choices with callbacks
storage/maria/ha_maria.cc:
Disable logging of index pages during repair
storage/maria/ma_bitmap.c:
Initialize callbacks explictily, not with pagecache_init(), to make things more readable and for future to have more choices with callbacks
Use new interface to flush logs from pagecache
storage/maria/ma_check.c:
Fixed test for wrong keyblocks
Use default functions to setup callbacks for pagecache
storage/maria/ma_loghandler.c:
Use dummy functions for log flush callback (NULL doesn't work anymore)
storage/maria/ma_open.c:
Initialize callbacks explictily, not with pagecache_init(), to make things more readable and for future to have more choices with callbacks
Prefix external functions with _ma_
storage/maria/ma_pagecache.c:
Use new simpler interface to flush logs if needed
storage/maria/ma_pagecache.h:
Changed interface to a faster, simpler one to flush logs.
Now we have a function that takes care of flushing logs, instead of a function to get lsn address
storage/maria/ma_pagecrc.c:
Add functions for flushing logs
storage/maria/ma_recovery.c:
Rename functions
storage/maria/maria_chk.c:
Use default functions to setup callbacks for pagecache
storage/maria/maria_def.h:
Prefixd global functions with _ma_
storage/maria/unittest/ma_pagecache_consist.c:
Use dummy functions for log flush callback (NULL doesn't work anymore)
storage/maria/unittest/ma_pagecache_single.c:
Use dummy functions for log flush callback (NULL doesn't work anymore)
storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
Use maria_flush_log_for_page to flush log pages. Fixes failure in unittest
mysql-test/include/maria_empty_logs.inc:
At one moment in maria-recovery.test the first log has number 2,
because log 1 was manually deleted.
mysql-test/r/maria-recovery.result:
after-merge fix
mysql-test/t/maria-recovery.test:
after-merge fix
storage/maria/ma_bitmap.c:
after-merge fix. The todo is implemented now.
storage/maria/ma_blockrec.c:
comment
storage/maria/ma_open.c:
after-merge fix. Set write_fail also for index file or a write error
would crash.
storage/maria/ma_pagecache.c:
comment
storage/maria/ma_pagecache.h:
I prefer to use NULL for 'no callback' instead of a dummy callback
in the special case of get_log_address; indeed for non-transactional
tables it uses an if(), while if using a dummy callback, it would
use a function call plus an if() (the dummy callback would need to
return a magic value to say "don't flush" and that value would be
tested in if()).
storage/maria/unittest/ma_test_all-t:
fix if running from outside storage/maria
* to honour WAL we now force the whole log when flushing a bitmap page.
* ability to intentionally crash in various places for recovery testing
* bugfix (dirty pages list found in checkpoint record was ignored)
* smaller checkpoint record
* misc small cleanups and comments
mysql-test/include/maria_empty_logs.inc:
maria-purge.test creates ~11 logs, remove them all
mysql-test/r/maria-recovery-bitmap.result:
result is good; without the _ma_bitmap_get_log_address() call,
we got
check error Bitmap at 0 has pages reserved outside of data file length
mysql-test/r/maria-recovery.result:
result update
mysql-test/t/maria-recovery-bitmap.test:
enable test of "bitmap-flush should flush whole log otherwise
corrupted data file (bitmap ahead of data pages)".
mysql-test/t/maria-recovery.test:
test of checkpoint
sql/sql_table.cc:
comment
storage/maria/ha_maria.cc:
_ma_reenable_logging_for_table() now includes file->trn=0.
At the end of repair() we don't need to re-enable logging, it is
done already by caller (like copy_data_between_tables()); it sounds
strange that this function could decide to re-enable, it should be
up to caller who knows what other operations it plans. Removing this
line led to assertion failure in maria_lock_database(F_UNLCK), fixed
by removing the assertion: maria_lock_database()
is here called in a context where F_UNLCK does not make the
table visible to others so assertion is excessive, and external_lock()
is already designed to honour the asserted condition.
Ability to crash at the end of bulk insert when indices
have been enabled.
storage/maria/ma_bitmap.c:
Better use pagecache_file_init() than set pagecache callbacks directly;
and a new function to set those callbacks for bitmap so that we can
reuse it.
_ma_bitmap_get_log_address() is a pagecache get_log_address callback
which causes the whole log to be flushed when a bitmap page
is flushed by the page cache. This was required by WAL.
storage/maria/ma_blockrec.c:
get_log_address pagecache callback for data (non bitmap) pages:
just reads the LSN from the page's content, like was hard-coded
before in ma_pagecache.c.
storage/maria/ma_blockrec.h:
functions which need to be exported
storage/maria/ma_check.c:
create_new_data_handle() can be static.
Ability to crash after rebuilding the index in OPTIMIZE,
in REPAIR. my_lock() implemented already.
storage/maria/ma_checkpoint.c:
As MARIA_SHARE* is now accessible to pagecache_collect_changed_blocks_LSN(),
we don't need to store kfile/dfile descriptors in checkpoint record,
2-byte-id of the table plus one byte to say if this is data or index
file is enough. So we go from 4+4 bytes per table down to 2+1.
storage/maria/ma_commit.c:
removing duplicate functions (see _ma_tmp_disable_logging_for_table())
storage/maria/ma_extra.c:
Monty fixed
storage/maria/ma_key_recover.c:
comment
storage/maria/ma_locking.c:
Sometimes other code does funny things with maria_lock_database(),
like ha_maria::repair() calling it at start and end without going
through ha_maria::external_lock(). So it happens that maria_lock_database()
is called with now_transactional!=born_transactional.
storage/maria/ma_loghandler.c:
update to new prototype
storage/maria/ma_open.c:
set_data|index_pagecache_callbacks() need to be exported as
they are now called when disabling/enabling transactionality.
storage/maria/ma_pagecache.c:
Removing PAGE_LSN_OFFSET, as much of the code relies on it being
0 anyway (let's not give impression we can just change this constant).
When flushing a page to disk, call the get_log_address callback to
know up to which LSN the log should be flushed.
As we now can access MARIA_SHARE* we can know share->id and store
it into the checkpoint record; we thus go from 4 bytes per dirty page
to 2+1.
storage/maria/ma_pagecache.h:
get_log_address callback
storage/maria/ma_panic.c:
No reason to reset pagecache callbacks in HA_PANIC_READ:
all we do is reopen files if they were closed; callbacks should
be in place already as 'info' exists; we just want to modify
the file descriptors, not the full PAGECACHE_FILE structure.
If we open data file and it was closed, share->bitmap.file needs
to be set.
Note that the modified code is disabled anyway.
storage/maria/ma_recovery.c:
Checkpoint record does not contain kfile/dfile descriptors anymore
so code can be simplified. Hash key in all_dirty_pages is
not made from file_descriptor & pageno anymore, but
index_or_data & table-short-id & pageno.
If a table's create_rename_lsn is higher than record's LSN,
we skip the table and don't fail if it's corrupted (because the LSNs
say that we don't have to look at this table).
If a table is skipped (for example due to create_rename_lsn),
its UNDOs still cause undo_lsn to advance; this is so that if later
we notice the transaction has to rollback we fail (as table should
not be skipped in this case).
Fixing a bug: the dirty_pages list was never used, because
the LSN below which it was used was the minimum rec_lsn of dirty pages!
It is now the min(checkpoint_start_log_horizon, min(trn's rec_lsn)).
When we disable/reenable transactionality, we modify pagecache
callbacks (needed for example for get_log_address: changing
share->page_type is not enough anymore).
storage/maria/ma_write.c:
'records' and 'checksum' are protected: they are updated under
log's mutex in write-hooks when UNDO is written.
storage/maria/maria_chk.c:
remove use of duplicate functions.
storage/maria/maria_def.h:
set_data|index_pagecache_callbacks() need to be exported;
_ma_reenable_logging_for_table() changes to a real function.
storage/maria/unittest/ma_pagecache_consist.c:
new prototype
storage/maria/unittest/ma_pagecache_single.c:
new prototype
storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
new prototype
Don't give warnings when opening a wrong control file in ma_control_file_t (unit test)
BitKeeper/deleted/.del-ma_key_redo.c:
Delete: storage/maria/ma_key_redo.c
storage/maria/ma_blockrec.c:
Don't give warnings when reading blocks when we know the block may not exist
storage/maria/ma_pagecache.c:
Keep a convinience copy of readwrite_flags
storage/maria/ma_pagecache.h:
Keep a convinience copy of readwrite_flags
storage/maria/unittest/ma_control_file-t.c:
Don't give warnings when opening a wrong control file
Added debugger hook _my_dbug_put_break_here() that is called if we get a CRC that matches --debug-crc-break (my_crc_dbug_break)
Fixed REDO_REPAIR to use all repair modes (repair, repair_by_sort, repair_paralell
REDO_REPAIR now also logs used key map
Fixed some bugs in REDO logging of key pages
Better error messages from maria_read_log
Added my_readwrite_flags to init_pagecache() to be able to get better error messages and simplify code.
Don't allow pagecaches with less than 8 blocks (Causes strange crashes)
Added EXTRA_DEBUG_KEY_CHANGES. When this is defined some REDO_INDEX entries contains page checksums (these are calculated and checked in DBUG mode, ignored otherwise)
Fixed bug in ma_pagecache unit tests that caused program to sometimes fail
Added some missing calls to MY_INIT() that caused some unit tests to fail
Fixed that TRUNCATE works properly on temporary MyISAM files
Updates some result files to new table checksums results (checksum when NULL fields are ignored)
perl test-insert can be replayed with maria_read_log!
sql/share/Makefile.am:
Change mode to -rw-rw-r--
BitKeeper/etc/ignore:
added storage/maria/unittest/page_cache_test_file_1 storage/maria/unittest/pagecache_debug.log
include/maria.h:
Added maria_tmpdir
include/my_base.h:
Added error HA_ERR_FILE_TOO_SHORT
include/my_sys.h:
Added variable my_crc_dbug_check
Added function my_dbug_put_break_here()
include/myisamchk.h:
Added org_key_map (Needed for writing REDO record for REPAIR)
mysql-test/r/innodb.result:
Updated to new checksum algorithm (NULL ignored)
mysql-test/r/mix2_myisam.result:
Updated to new checksum algorithm (NULL ignored)
mysql-test/r/myisam.result:
Updated to new checksum algorithm (NULL ignored)
mysql-test/t/myisam.test:
Added used table
mysys/checksum.c:
Added DBUG for checksum results
Added debugger hook so that _my_dbug_put_break_here() is called if we get matching CRC
mysys/lf_alloc-pin.c:
Fixed compiler warning
mysys/my_handler.c:
Added new error message
mysys/my_init.c:
If my_progname is not given, use 'unknown' form my_progname_short
Added debugger function my_debug_put_break_here()
mysys/my_pread.c:
In case of too short file when MY_NABP or MY_FNABP is specified, give error HA_ERR_FILE_TO_SHORT
mysys/my_read.c:
In case of too short file when MY_NABP or MY_FNABP is specified, give error HA_ERR_FILE_TO_SHORT
sql/mysqld.cc:
Added debug option --debug-crc-break
sql/sql_parse.cc:
Trivial optimization
storage/maria/ha_maria.cc:
Renamed variable to be more logical
Ensure that param.testflag is correct when calling repair
Added extra argument to init_pagecache
Set default value for maria_tempdir
storage/maria/ma_blockrec.c:
Test for HA_ERR_FILE_TOO_SHORT instead for -1
storage/maria/ma_cache.c:
Test for HA_ERR_FILE_TOO_SHORT instead for -1
storage/maria/ma_check.c:
Set param->testflag to match how repair is run (needed for REDO logging)
Simple optimization
Moved flag if page is node from pagelength to keypage-flag byte
Log used key map in REDO log.
storage/maria/ma_delete.c:
Remember previous UNDO entry when writing undo (for future CLR records)
Moved flag if page is node from pagelength to keypage-flag byte
Fixed some bugs in redo logging
Added CRC for some translog REDO_INDEX entries
storage/maria/ma_dynrec.c:
Test for HA_ERR_FILE_TOO_SHORT instead for -1
storage/maria/ma_ft_update.c:
Fixed call to _ma_store_page_used()
storage/maria/ma_key_recover.c:
Added CRC for some translog REDO_INDEX entries
Removed not needed pagecache_write() in _ma_apply_redo_index()
storage/maria/ma_locking.c:
Test for HA_ERR_FILE_TOO_SHORT instead for -1
storage/maria/ma_loghandler.c:
Added used key map to REDO_REPAIR_TABLE
storage/maria/ma_loghandler.h:
Added operation for checksum of key pages
storage/maria/ma_open.c:
Allocate storage for undo lsn pointers
storage/maria/ma_pagecache.c:
Remove not needed include file
Change logging to use fd: for file descritors as other code
Added my_readwrite_flags to init_pagecache() to be able to get better error messages for maria_chk/maria_read_log
Don't allow pagecaches with less than 8 blocks
Remove wrong DBUG_ASSERT()
storage/maria/ma_pagecache.h:
Added readwrite_flags
storage/maria/ma_recovery.c:
Better error messages for maria_read_log:
- Added eprint() for printing error messages
- Print extra \n before error message if we are printing %0 %10 ...
Added used key_map to REDO_REPAIR log entry
More DBUG
Call same repair method that was used by mysqld
storage/maria/ma_rt_index.c:
Moved flag if page is node from pagelength to keypage-flag byte
storage/maria/ma_rt_key.c:
Fixed call to _ma_store_page_used()
storage/maria/ma_rt_split.c:
Moved flag if page is node from pagelength to keypage-flag byte
storage/maria/ma_static.c:
Added maria_tmpdir
storage/maria/ma_test1.c:
Updated call to init_pagecache()
storage/maria/ma_test2.c:
Updated call to init_pagecache()
storage/maria/ma_test3.c:
Updated call to init_pagecache()
storage/maria/ma_write.c:
Removed #ifdef NOT_YET
Moved flag if page is node from pagelength to keypage-flag byte
Fixed bug in _ma_log_del_prefix()
storage/maria/maria_chk.c:
Fixed wrong min limit for page_buffer_size
Updated call to init_pagecache()
storage/maria/maria_def.h:
Added EXTRA_DEBUG_KEY_CHANGES. When this is defined some REDO_INDEX entries contains page checksums
Moved flag if page is node from pagelength to keypage-flag byte
storage/maria/maria_ftdump.c:
Updated call to init_pagecache()
storage/maria/maria_pack.c:
Updated call to init_pagecache()
Reset share->state.create_rename_lsn & share->state.is_of_horizon
storage/maria/maria_read_log.c:
Better error messages
Added --tmpdir option (needed to set temporary directory for REDO_REPAIR)
Added --start-from-lsn
Changed option for --display-only to 'd' (wanted to use -o for 'offset')
storage/maria/unittest/lockman2-t.c:
Added missing call to MY_INIT()
storage/maria/unittest/ma_pagecache_consist.c:
Updated call to init_pagecache()
storage/maria/unittest/ma_pagecache_single.c:
Fixed bug that caused program to sometimes fail
Added some DBUG_ASSERTS()
Changed some calls to malloc()/free() to my_malloc()/my_free()
Create extra file to expose original hard-to-find bug
storage/maria/unittest/ma_test_loghandler-t.c:
Updated call to init_pagecache()
storage/maria/unittest/ma_test_loghandler_first_lsn-t.c:
Updated call to init_pagecache()
storage/maria/unittest/ma_test_loghandler_max_lsn-t.c:
Updated call to init_pagecache()
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
Updated call to init_pagecache()
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
Updated call to init_pagecache()
storage/maria/unittest/ma_test_loghandler_noflush-t.c:
Updated call to init_pagecache()
storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
Updated call to init_pagecache()
storage/maria/unittest/ma_test_loghandler_purge-t.c:
Updated call to init_pagecache()
storage/maria/unittest/test_file.c:
Changed malloc()/free() to my_malloc()/my_free()
Fixed memory leak
Changd logic a bit while trying to find bug in reset_file()
storage/maria/unittest/trnman-t.c:
Added missing call to MY_INIT()
storage/myisam/mi_cache.c:
Test for HA_ERR_FILE_TOO_SHORT instead for -1
storage/myisam/mi_create.c:
Removed O_EXCL to get TRUNCATE to work for temporary files
storage/myisam/mi_dynrec.c:
Test for HA_ERR_FILE_TOO_SHORT instead for -1
storage/myisam/mi_locking.c:
Test for HA_ERR_FILE_TOO_SHORT instead for -1
mysql-test/r/old-mode.result:
New BitKeeper file ``mysql-test/r/old-mode.result''
mysql-test/t/old-mode-master.opt:
New BitKeeper file ``mysql-test/t/old-mode-master.opt''
mysql-test/t/old-mode.test:
New BitKeeper file ``mysql-test/t/old-mode.test''
into mysql.com:/home/my/mysql-maria
include/my_sys.h:
Auto merged
sql/mysqld.cc:
Auto merged
storage/maria/ma_checkpoint.c:
Auto merged
storage/maria/ma_pagecache.c:
Auto merged
storage/maria/ma_pagecache.h:
Auto merged
storage/maria/maria_chk.c:
Auto merged
storage/maria/ma_recovery.c:
SCCS merged
New extendable format for maria_log_control file
Fixed some compiler warnings
include/maria.h:
Added maria_disable_logging() and maria_enable_logging()
mysql-test/include/maria_verify_recovery.inc:
Updated tests now when key redo/undo works
mysql-test/r/maria-recovery.result:
Updated tests now when key redo/undo works
storage/maria/ma_blockrec.c:
Use unified CLR code
Added rec_lsn for full pages
Moved clr write hook to ma_key_recover.c
Changed REDO code to keep pages pinned until undo
Mark page_link's as changed
storage/maria/ma_blockrec.h:
Moved write_hook_for_clr_end() to ma_key_recover.c
storage/maria/ma_check.c:
Changed key check code to use PAGECACHE_READ_UNKNOWN_PAGE
Fixed wrong warning when checking files after maria_pack
When unpacking files, we have to use new keypos_to_recpos method
When doing repair, we can disregard index key file pages in page cache
storage/maria/ma_commit.c:
Added simple enable/disable logging functions
(Needed for recovery)
storage/maria/ma_control_file.c:
Make maria control file extendable without having to make it incompatible for older versions
storage/maria/ma_control_file.h:
New error messages
Added CONTROL_FILE_VERSION
storage/maria/ma_delete.c:
Added redo/undo for key pages
change_length -> changed_length to make things similar
More comments & more DBUG
storage/maria/ma_key_recover.c:
Unified CLR method
Moved here write_hook_for_clr_end() and common keypage log functions
Changed REDO to keep pages pinned until undo
Changed UNDO code to change key_root under log mutex
storage/maria/ma_key_recover.h:
New structures and functions
storage/maria/ma_loghandler.c:
Include needed files
storage/maria/ma_open.c:
Change maria_open() to use pread() instead of read()
storage/maria/ma_page.c:
Fixed bug in key_del handling
Clear pages if IDENTICAL_PAGES_AFTER_RECOVERY is defined
storage/maria/ma_pagecache.c:
Indentation and spelling fixes
More DBUG
Added helper function: pagecache_block_link_to_buffer()
storage/maria/ma_pagecache.h:
Added pagecache_block_link_to_buffer()
storage/maria/ma_recovery.c:
Fixed state.changed
Fixed that REDO keeps pages pinned until UNDO
Some bug fixes from previous commit
Fixes for UNDO/REDO of key pages
storage/maria/ma_search.c:
Fixed packing and storing of keys to provide more information to caller so
that we can do efficent REDO logging of the changes.
storage/maria/ma_test1.c:
Fixed bug with not initialized variable
storage/maria/ma_test2.c:
Removed not used code
storage/maria/ma_test_all.res:
Updated results
storage/maria/ma_test_all.sh:
Changed one test to test more
Removed timing tests as not relevant here
storage/maria/ma_test_recovery.expected:
Updated test result after redo/undo if key pages works
storage/maria/ma_test_recovery:
Updated test after redo/undo if key pages works
storage/maria/ma_write.c:
Moved some general log functions to ma_key_recover.c
Fixed some bugs in undo
Moved ma_log_split() to _ma_split_page()
Small changes in some function arguments to be able to do redo logging
storage/maria/maria_chk.c:
disable logging while doing repair table
storage/maria/maria_def.h:
New function prototypes
Move some structs and functions to ma_key_recover.c
storage/maria/unittest/ma_control_file-t.c:
Updated with patch from Sanja
NOTE: This is not complete and need to be updated to new control file format
storage/maria/unittest/ma_test_loghandler-t.c:
Fixed compiler warning
into mysql.com:/home/my/mysql-maria
storage/maria/ha_maria.cc:
Auto merged
storage/maria/ma_bitmap.c:
Auto merged
storage/maria/ma_checkpoint.c:
Auto merged
storage/maria/ma_close.c:
Auto merged
storage/maria/ma_loghandler.c:
Auto merged
storage/maria/ma_loghandler.h:
Auto merged
storage/maria/ma_open.c:
Auto merged
storage/maria/ma_pagecache.h:
Auto merged
storage/maria/ma_write.c:
Auto merged
storage/maria/maria_def.h:
Auto merged
storage/maria/unittest/ma_pagecache_single.c:
Auto merged
storage/maria/ma_blockrec.c:
Manual merge
storage/maria/ma_page.c:
Manual merge
storage/maria/ma_pagecache.c:
Manual merge
storage/maria/ma_preload.c:
Manual merge
storage/maria/ma_recovery.c:
Manual merge
Add _ma_unpin_all_pages() to all new UNDO redo_exec_hook's
Added key_nr to st_maria_keydef for faster keyinfo->keynr conversion
For transactional tables, shift record number in keys up with 1 bit to have place to indicate if transid follows
Checksum for MyISAM now ignores NULL and not used part of VARCHAR
Renamed some variables that caused shadow compiler warnings
Moved extra() call when waiting for tables to not be used to after tables are removed from cache.
Fixed crashing bugs when using Maria TEMPORARY tables with TRUNCATE. Removed 'hack' code in sql directory to go around this bug.
pagecache_unlock_by_ulink() now has extra argument to say if page was changed.
Give error message if we fail to open control file
Mark page cache variables as not flushable
include/maria.h:
Made min page cache larger (needed for pinning key page)
Added key_nr to st_maria_keydef for faster keyinfo->keynr conversion
Added write_comp_flag to move some runtime code to maria_open()
include/my_base.h:
Added new error message to be used when handler initialization failed
include/my_global.h:
Renamed dummy to swap_dummy to avoid conflicts with local 'dummy' variables
include/my_handler.h:
Added const to some parameters
mysys/array.c:
More DBUG
mysys/my_error.c:
Fixed indentation
mysys/my_handler.c:
Added const to some parameters
Added missing error messages
sql/field.h:
Renamed variables to avoid variable shadowing
sql/handler.h:
Renamed parameter to avoid variable name conflict
sql/item.h:
Renamed variables to avoid variable shadowing
sql/log_event_old.h:
Renamed variables to avoid variable shadowing
sql/set_var.h:
Renamed variables to avoid variable shadowing
sql/sql_delete.cc:
Removed maria hack for temporary tables
Fixed indentation
sql/sql_table.cc:
Moved extra() call when waiting for tables to not be used to after tables are removed from cache.
This was needed to ensure we don't do a PREPARE_FOR_DROP or similar call while the table is still in use.
sql/table.cc:
Copy page_checksum from share
Removed Maria hack
storage/maria/Makefile.am:
Added new files
storage/maria/ha_maria.cc:
Renamed records -> record_count and info -> create_info to avoid variable name conflicts
Mark page cache variables as not flushable
storage/maria/ma_blockrec.c:
Moved _ma_unpin_all_pages() to ma_key_recover.c
Moved init of info->pinned_pages to ma_open.c
Moved _ma_finalize_row() to maria_key_recover.h
Renamed some variables to avoid variable name conflicts
Mark page_link.changed for blocks we change directly
Simplify handling of undo link when writing LOGREC_UNDO_ROW_INSERT (old code crashed when having redo for index)
storage/maria/ma_blockrec.h:
Removed extra empty line
storage/maria/ma_checkpoint.c:
Remove not needed trnman.h
storage/maria/ma_close.c:
Free pinned pages (which are now always allocated)
storage/maria/ma_control_file.c:
Give error message if we fail to open control file
storage/maria/ma_delete.c:
Changes for redo logging (first part, logging of underflow not yet done)
- Log undo-key-delete
- Log delete of key
- Updated arguments to _ma_fetch_keypage(), _ma_dispose(), _ma_write_keypage(), _ma_insert()
- Added new arguments to some functions to be able to write redo information
- Mark key pages as changed when we write with PAGECACHE_LOCK_LEFT_WRITELOCKED
Remove one not needed _ma_write_keypage() in d_search() when upper level will do the write anyway
Changed 2 bmove_upp() to bmove() as this made code easer to understand
More function comments
Indentation fixes
storage/maria/ma_ft_update.c:
New arguments to _ma_write_keypage()
storage/maria/ma_loghandler.c:
Fixed some DBUG_PRINT messages
Simplify code
Added new log entrys for key page redo
Renamed some variables to avoid variable name shadowing
storage/maria/ma_loghandler.h:
Moved some defines here
Added define for storing key number on key pages
Added new translog record types
Added enum for type of operations in LOGREC_REDO_INDEX
storage/maria/ma_open.c:
Always allocate info.pinned_pages (we need now also for normal key page usage)
Update keyinfo->key_nr
Added virtual functions to convert record position o number to be stored on key pages
Update keyinfo->write_comp_flag to value of search flag to be used when writing key
storage/maria/ma_page.c:
Added redo for key pages
- Extended _ma_fetch_keypage() with type of lock to put on page and address to used MARIA_PINNED_PAGE
- _ma_fetch_keypage() now pin's pages if needed
- Extended _ma_write_keypage() with type of locks to be used
- ma_dispose() now locks info->s->state.key_del from other threads
- ma_dispose() writes redo log record
- ma_new() locks info->s->state.key_del from other threads if it was used
- ma_new() now pins read page
Other things:
- Removed some not needed arguments from _ma_new() and _ma_dispose)
- Added some new variables to simplify code
- If EXTRA_DEBUG is used, do crc on full page to catch not unitialized bytes
storage/maria/ma_pagecache.h:
Applied patch from Sanja to add extra argument to pagecache_unlock_by_ulink() to mark if page was changed
Added some defines for pagecache priority levels that one can use
storage/maria/ma_range.c:
Added new arguments for call to _ma_fetch_keypage()
storage/maria/ma_recovery.c:
- Added hooks for new translog types:
REDO_INDEX, REDO_INDEX_NEW_PAGE, REDO_INDEX_FREE_PAGE, UNDO_KEY_INSERT, UNDO_KEY_DELETE and
UNDO_KEY_DELETE_WITH_ROOT.
- Moved variable declarations to start of function (portability fixes)
- Removed some not needed initializations
- Set only relevant state changes for each redo/undo entry
storage/maria/lockman.c:
Removed end space
storage/maria/ma_check.c:
Removed end space
storage/maria/ma_create.c:
Removed end space
storage/maria/ma_locking.c:
Removed end space
storage/maria/ma_packrec.c:
Removed end space
storage/maria/ma_pagecache.c:
Removed end space
storage/maria/ma_panic.c:
Removed end space
storage/maria/ma_rt_index.c:
Added new arguments for call to _ma_fetch_keypage(), _ma_write_keypage(), _ma_dispose() and _ma_new()
Fixed indentation
storage/maria/ma_rt_key.c:
Added new arguments for call to _ma_fetch_keypage()
storage/maria/ma_rt_split.c:
Added new arguments for call to _ma_new()
Use new keypage header
Added new arguments for call to _ma_write_keypage()
storage/maria/ma_search.c:
Updated comments & indentation
Added new arguments for call to _ma_fetch_keypage()
Made some variables and arguments const
Added virtual functions for converting row position to number to be stored in key
use MARIA_RECORD_POS of record position instead of my_off_t
Record in MARIA_KEY_PARAM how page was changed one key insert (needed for REDO)
storage/maria/ma_sort.c:
Removed end space
storage/maria/ma_statrec.c:
Updated arguments for call to _ma_rec_pos()
storage/maria/ma_test1.c:
Fixed too small buffer to init_pagecache()
Fixed bug when using insert_count and test_flag
storage/maria/ma_test2.c:
Use more resonable pagecache size
Remove not used code
Reset blob_length to fix wrong output message
storage/maria/ma_test_all.sh:
Fixed wrong test
storage/maria/ma_write.c:
Lots of new code to handle REDO of key pages
No logic changes because of REDO code, mostly adding new arguments and adding new code for logging
Added new arguments for calls to _ma_fetch_keypage(), _ma_write_keypage() and similar functions
Move setting of comp_flag in ma_ck_wrte_btree() from runtime to maria_open()
Zerofill new used pages for:
- To remove possible sensitive data left in buffer
- To get idenitical data on pages after running redo
- Better compression of pages if archived
storage/maria/maria_chk.c:
Added information if table is crash safe
storage/maria/maria_def.h:
New virtual function to convert between record position on key and normal record position
Aded mutex and extra variables to handle locking of share->state.key_del
Moved some structure variables to get things more aligned
Added extra arguments to MARIA_KEY_PARAM to be able to remember what was changed on key page on key insert
Added argument to MARIA_PINNED_PAGE to indicate if page was changed
Updated prototypes for functions
Added some structures for signaling changes in REDO handling
storage/maria/unittest/ma_pagecache_single.c:
Updated arguments for changed function calls
storage/myisam/mi_check.c:
Made calc_check_checksum virtual
storage/myisam/mi_checksum.c:
Update checksums to ignore null columns
storage/myisam/mi_create.c:
Mark if table has null column (to know when we have to use mi_checksum())
storage/myisam/mi_open.c:
Added virtual function for calculating checksum to be able to easily ignore NULL fields
storage/myisam/mi_test2.c:
Fixed bug
storage/myisam/myisamdef.h:
Added virtual function for calculating checksum during check table
Removed ha_key_cmp() as this is in handler.h
storage/maria/ma_key_recover.c:
New BitKeeper file ``storage/maria/ma_key_recover.c''
storage/maria/ma_key_recover.h:
New BitKeeper file ``storage/maria/ma_key_recover.h''
storage/maria/ma_key_redo.c:
New BitKeeper file ``storage/maria/ma_key_redo.c''
- serializing calls to flush_pagecache_blocks_int() on the same file
to avoid known concurrency bugs
- having that, we can now enable the background thread, as the
flushes it does are now supposedly safe in concurrent situations.
- new type of flush FLUSH_KEEP_LAZY: when the background checkpoint
thread is flushing a packet of dirty pages between two checkpoints,
it uses this flush type, indeed if a file is already being flushed
by another thread it's smarter to move on to the next file than wait.
- maria_checkpoint_frequency renamed to maria_checkpoint_interval.
include/my_sys.h:
new type of flushing for the page cache: FLUSH_KEEP_LAZY
mysql-test/r/maria.result:
result update
mysys/mf_keycache.c:
indentation. No FLUSH_KEEP_LAZY support in key cache.
storage/maria/ha_maria.cc:
maria_checkpoint_frequency was somehow a hidden part of the
Checkpoint API and that was not good. Now we have checkpoint_interval,
local to ha_maria.cc, which serves as container for the user-visible
maria_checkpoint_interval global variable; setting it calls
update_checkpoint_interval which passes the new value to
ma_checkpoint_init(). There is no hiding anymore.
By default, enable background thread which does checkpoints
every 30 seconds, and dirty page flush in between. That thread takes
a checkpoint when it ends, so no need for maria_hton_panic to take one.
The | is | and not ||, because maria_panic() must always be called.
frequency->interval.
storage/maria/ma_checkpoint.c:
Use FLUSH_KEEP_LAZY for background thread when it flushes packets of
dirty pages between two checkpoints: it is smarter to move on to
the next file than wait for it to have been completely flushed, which
may take long.
Comments about flush concurrency bugs moved from ma_pagecache.c.
Removing out-of-date comment.
frequency->interval.
create_background_thread -> (interval>0).
In ma_checkpoint_background(), some variables need to be preserved
between iterations.
storage/maria/ma_checkpoint.h:
new prototype
storage/maria/ma_pagecache.c:
- concurrent calls of flush_pagecache_blocks_int() on the same file
cause bugs (see @note in that function); we fix them by serializing
in this situation. For that we use a global hash of (file, wqueue).
When flush_pagecache_blocks_int() starts it looks into the hash,
using the file as key. If not found, it inserts (file,wqueue) into the
hash, flushes the file, and finally removes itself from the hash and
wakes up any waiter in the queue. If found, it adds itself to the
wqueue and waits.
- As a by-product, we can remove changed_blocks_is_incomplete
and replace it by scanning the hash, replace the sleep() by a queue wait.
- new type of flush FLUSH_KEEP_LAZY: when flushing a file, if it's
already being flushed by another thread (even partially), return
immediately.
storage/maria/ma_pagecache.h:
In pagecache, a hash of files currently being flushed (i.e. there
is a call to flush_pagecache_blocks_int() for them).
storage/maria/ma_recovery.c:
new prototype
storage/maria/ma_test1.c:
new prototype
storage/maria/ma_test2.c:
new prototype
Ability for flush_pagecache_blocks() to flush only certain pages of
a file, as instructed by an option "filter" pointer-to-function argument;
Checkpoint and background dirty page flushing use that to flush only
pages which have been dirty for long enough and bitmap pages.
Fix for a bug in flush_cached_blocks() (no idea if it could produce
a bug in real life, but theoretically it is).
Testing checkpoint in ma_test_recovery via ma_test1 and ma_test2.
Background checkpoint & dirty pages flush thread is still disabled
by default in ha_maria.
mysql-test/r/maria.result:
result update
storage/maria/ha_maria.cc:
blank after function comment
storage/maria/ma_checkpoint.c:
Using an enum instead of 0/1/2 (applying Sanja's review comments).
The comment about "this is an horizon" can be removed as Sanja
created translog_next_LSN() which parse_checkpoint_record() uses.
Variables in ma_checkpoint_background() cannot be declared in the
for() as their value must not be reset at each iteration!
storage/maria/ma_pagecache.c:
adding to flush_pagecache_blocks() optional arguments 'filter'
(pointer to function) and 'filter_arg'; if filter!=NULL this function
will be called for each block of the file and will reply if this
block and following ones should be flushed or not (3 possible
replies).
Fixing a bug when flush_cached_blocks() skips a pinned page: it has
to unset PCBLOCK_IN_FLUSH set by flush_pagecache_blocks_int().
storage/maria/ma_pagecache.h:
flush_pagecache_blocks() is changed to take "filter" and "filter_arg"
arguments. "filter", if it is not NULL, may return one value
among enum pagecache_flush_filter_result.
storage/maria/ma_recovery.c:
open_count=0 when closing tables at the end of recovery.
storage/maria/ma_test1.c:
Optional checkpoints (-H#) at various stages (stages similar
to --testflag), for testing of checkpoints.
storage/maria/ma_test2.c:
Optional checkpoints (-H#) at various stages (stages similar
to -t), for testing of checkpoints.
storage/maria/ma_test_recovery.expected:
Result update: the results of the additional test run with -H#
(checkpoints) are added here. They are exactly identical to without
checkpoints except that the index's Root (printed by maria_chk)
is more correct when using checkpoints. This is because checkpoint
flushed the state, so it happens to be correct, while no-checkpoint
does not flush the state, and recovery does not recover indexes
so Root is never fixed. When we recover indices, this will go away.
storage/maria/ma_test_recovery:
We duplicate the loop of tests to add an additional run with
checkpoints at various stages, to see if maria_read_log
uses them fine.
REDO optimization (Bascily avoid moving blocks from/to pagecache)
More command line arguments to maria_read_log
Fixed recovery bug when recreating table
sql/opt_range.cc:
Remove SAFE_MODE for opt_range as it disables UPDATE to use keys
storage/maria/ma_blockrec.c:
REDO optimization
Use new interface for pagecache_reads to avoid copying page buffers
storage/maria/ma_loghandler.c:
Patch from Sanja:
- Added new parameter to translog_get_page to use direct links to pagecache
- Changed scanner to be able to use direct links
This avoids a lot of calls to bmove512() in page cache.
storage/maria/ma_loghandler.h:
Added direct link to pagecache objects
storage/maria/ma_open.c:
Added const to parameter
Added missing braces
storage/maria/ma_pagecache.c:
From Sanja:
- Added direct links to pagecache (from pagecache_read())
Dirrect link means that on pagecache_read we get back a pointer to the pagecache buffer
From Monty:
- Fixed arguments to init_page_cache to handle big page caches
- Fixed compiler warnings
- Replaced PAGECACHE_PAGE_LINK with PAGECACHE_BLOCK_LINK * to catch errors
storage/maria/ma_pagecache.h:
Changed block numbers from int to long to be able to handle big page caches
Changed some PAGECACHE_PAGE_LINK to PAGECACHE_BLOCK_LINK
storage/maria/ma_recovery.c:
Fixed recovery bug when recreating table (table was kept open)
Moved some variables to function start (portability)
Added space to some print messages
storage/maria/maria_chk.c:
key_buffer_size -> page_buffer_size
storage/maria/maria_def.h:
Changed default page_buffer_size to 10M
storage/maria/maria_read_log.c:
Added more startup options:
--version
--undo (apply undo)
--page_cache_size (to run with big cache sizes)
--silent (to not get any output from --apply)
storage/maria/unittest/ma_control_file-t.c:
Fixed compiler warning
storage/maria/unittest/ma_test_loghandler-t.c:
Added new argument to translog_init_scanner()
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
Added new argument to translog_init_scanner()
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
Added new argument to translog_init_scanner()
Finally this is the real checkpoint code.
It however exhibits unstabilities when a checkpoint runs concurrently
with data-modifying clients (table corruption, transaction log's
assertions) so for now a checkpoint is taken only at startup after
recovery and at shutdown, i.e. not in concurrent situations. Later
we will let it run periodically, as well as flush dirty pages
periodically (almost all needed code is there already, only pagecache
code is written but not committed).
WL#3072 Maria recovery
* replacing UNDO_ROW_PURGE with CLR_END; testing of those CLR_END via
ma_test2 which has INSERTs failing with duplicate keys.
* replaying of REDO_RENAME_TABLE
Now, off to test Recovery in ha_maria :)
BitKeeper/deleted/.del-ma_least_recently_dirtied.c:
Delete: storage/maria/ma_least_recently_dirtied.c
BitKeeper/deleted/.del-ma_least_recently_dirtied.h:
Delete: storage/maria/ma_least_recently_dirtied.h
storage/maria/Makefile.am:
compile Checkpoint module
storage/maria/ha_maria.cc:
When ha_maria starts, do a recovery from last checkpoint.
Take a checkpoint when that recovery has ended and when ha_maria
shuts down cleanly.
storage/maria/ma_blockrec.c:
* even if my_sync() fails we have to my_close() (otherwise we leak
a descriptor)
* UNDO_ROW_PURGE is replaced by a simple CLR_END for UNDO_ROW_INSERT,
as promised in the old comment; it gives us skipping during the
UNDO phase.
storage/maria/ma_check.c:
All REDOs before create_rename_lsn are ignored by Recovery. So
create_rename_lsn must be set only after all data/index has been
flushed and forced to disk. We thus move write_log_record_for_repair()
to after _ma_flush_tables_files_after_repair().
storage/maria/ma_checkpoint.c:
Checkpoint module.
storage/maria/ma_checkpoint.h:
optional argument if caller wants a thread to periodically take
checkpoints and flush dirty pages.
storage/maria/ma_create.c:
* no need to init some vars as the initial bzero(share) takes care of this.
* update to new function's name
* even if we fail in my_sync() we have to my_close()
storage/maria/ma_extra.c:
Checkpoint reads share->last_version under intern_lock, so we make
maria_extra() update it under intern_lock. THR_LOCK_maria still needed
because of _ma_test_if_reopen().
storage/maria/ma_init.c:
destroy checkpoint module when Maria shuts down.
storage/maria/ma_loghandler.c:
* UNDO_ROW_PURGE gone (see ma_blockrec.c)
* we need to remember the LSN of the LOGREC_FILE_ID for a share,
because this LSN is needed into the checkpoint record (Recovery wants
to know the validity domain of an id->name mapping)
* translog_get_horizon_no_lock() needed for Checkpoint
* comment about failing assertion (Sanja knows)
* translog_init_reader_data() thought that translog_read_record_header_scan()
returns 0 in case of error, but 0 just means "0-length header".
* translog_assign_id_to_share() now needs the MARIA_HA because
LOGREC_FILE_ID uses a log-write hook.
* Verify that (de)assignment of share->id happens only under intern_lock,
as Checkpoint reads this id with intern_lock.
* translog_purge() can accept TRANSLOG_ADDRESS, not necessarily
a real LSN.
storage/maria/ma_loghandler.h:
prototype updates
storage/maria/ma_open.c:
no need to initialize "res"
storage/maria/ma_pagecache.c:
When taking a checkpoint, we don't need to know the maximum rec_lsn
of dirty pages; this LSN was intended to be used in the two-checkpoint
rule, but last_checkpoint_lsn is as good.
4 bytes for stored_list_size is enough as PAGECACHE::blocks (number
of blocks which the pagecache can contain) is int.
storage/maria/ma_pagecache.h:
new prototype
storage/maria/ma_recovery.c:
* added replaying of REDO_RENAME_TABLE
* UNDO_ROW_PURGE gone (see ma_blockrec.c), replaced by CLR_END
* Recovery from the last checkpoint record now possible
* In new_table() we skip the table if the id->name mapping is older than
create_rename_lsn (mapping dates from lsn_of_file_id).
* in get_MARIA_HA_from_REDO_record() we skip the record
if the id->name mapping is newer than the record (can happen if processing
a record which is before the checkpoint record).
* parse_checkpoint_record() has to return a LSN, that's what caller expects
storage/maria/ma_rename.c:
new function's name; log end zeroes of tables' names (ease recovery)
storage/maria/ma_test2.c:
* equivalent of ma_test1's --test-undo added (named -u here).
* -t=1 now stops right after creating the table, so that
we can test undoing of INSERTs with duplicate keys (which tests the
CLR_END logged by _ma_write_abort_block_record()).
storage/maria/ma_test_recovery.expected:
Result of testing undoing of INSERTs with duplicate keys; there are
some differences in maria_chk -dvv but they are normal (removing
records does not shrink data/index file, does not put back the
"analyzed, optimized keys"(etc) index state.
storage/maria/ma_test_recovery:
Test undoing of INSERTs with duplicate keys, using ma_test2;
when such INSERT happens, it logs REDO_INSERT, UNDO_INSERT, REDO_DELETE,
CLR_END; we abort after that, and test that CLR_END causes recovery
to jump over UNDO_INSERT.
storage/maria/ma_write.c:
comment
storage/maria/maria_chk.c:
comment
storage/maria/maria_def.h:
* a new bit in MARIA_SHARE::in_checkpoint, used to build a list
of unique shares during Checkpoint.
* MARIA_SHARE::lsn_of_file_id added: the LSN of the last LOGREC_FILE_ID
for this share; needed to know to which LSN domain the mappings
found in the Checkpoint record apply (new mappings should not apply
to old REDOs).
storage/maria/trnman.c:
* small changes to how trnman_collect_transactions() fills its buffer;
it also uses a non-dummy lsn_read_non_atomic() found in ma_checkpoint.h
into desktop.sanja.is.com.ua:/home/bell/mysql/bk/work-maria
storage/maria/ma_pagecache.c:
Auto merged
storage/maria/unittest/ma_test_loghandler-t.c:
Auto merged
storage/maria/ma_loghandler.c:
merge
storage/maria/ma_pagecache.h:
merge
Already written pages injection to the cache fixed.
storage/maria/ma_loghandler.c:
Ability to read unflushed data added.
storage/maria/ma_page.c:
Parameters added
storage/maria/ma_pagecache.c:
Already written pages injection to the cache fixed.
Validator for case of page content injection added.
storage/maria/ma_pagecache.h:
Validator for case of page content injection added.
storage/maria/unittest/Makefile.am:
Test of reading unflushed data
storage/maria/unittest/ma_test_loghandler-t.c:
Define fixed.
Restart of the log removed.
storage/maria/unittest/ma_test_loghandler_noflush-t.c:
New BitKeeper file ``storage/maria/unittest/ma_test_loghandler_noflush-t.c''
found in pushbuild on sapsrv1. Some not fixed as not repeatable
on my machine (32/64 bit issue?).
Fixes for some test failures:
- "maria-connect" now passes;
- "maria": after fixing the obvious reasons for failures, the test
went further and hit a more complex issues: difference in the output
of EXPLAIN output; not fixed;
- "ps_maria" still crashes in assertion
mysqld: ha_maria.cc:1627: virtual int ha_maria::index_read(uchar*, const uchar*, uint, ha_rkey_function): Ass
ertion `inited == INDEX' failed, as already observable in pushbuild.
All this might just be due to an incomplete merge of MyISAM changes
into Maria when 5.1 was last merged to mysql-maria.
include/my_global.h:
temporary fix until next merge of 5.1; without this it does not build
mysql-test/r/maria-connect.result:
position changed
mysql-test/t/maria-connect.test:
If one wants to use the binlog it has to ask for it.
1582 is not used for dup entry error anymore (it was in older 5.1).
Size of first event in binlog was increased by 4 (when the new type
of event "gap" was added).
mysql-test/t/maria.test:
1582 not used anymore in this case
storage/maria/ha_maria.cc:
engine now has to say what binlogging it supports
storage/maria/ma_blockrec.c:
fix for compiler warnings ("comparison is always true" or "always
false")
storage/maria/ma_loghandler.c:
fix for compiler warnings (comparing char* to uchar*)
storage/maria/ma_packrec.c:
fix for compiler warning (fix simply merged from MyISAM)
storage/maria/ma_pagecache.c:
info_check_pin() was not used so gave a compiler warning.
storage/maria/ma_pagecache.h:
fixing typo from the last 5.1->maria merge.
storage/maria/ma_recovery.c:
my_free() has a void* argument, so why cast. byte->uchar.
storage/maria/ma_search.c:
fix for compiler warning (fix simply merged from MyISAM)
storage/maria/maria_read_log.c:
gptr->uchar*
storage/maria/trnman.c:
probable fix for warning found in pushbuild (but not on my machine):
storage/maria/trnman.c: 142
passing argument 6 of \u2018lf_hash_init\u2019 from incompatible pointer type
on sapsrv1.
into a88-113-38-195.elisa-laajakaista.fi:/home/my/bk/mysql-maria.prod
BitKeeper/etc/ignore:
auto-union
include/maria.h:
Auto merged
include/my_base.h:
Auto merged
mysql-test/include/ps_conv.inc:
Auto merged
mysql-test/r/ps_2myisam.result:
Auto merged
mysql-test/r/ps_3innodb.result:
Auto merged
mysql-test/r/ps_4heap.result:
Auto merged
mysql-test/r/ps_5merge.result:
Auto merged
mysql-test/r/ps_7ndb.result:
Auto merged
mysys/Makefile.am:
Auto merged
mysys/mf_iocache.c:
Auto merged
mysys/mf_tempfile.c:
Auto merged
mysys/my_error.c:
Auto merged
mysys/my_init.c:
Auto merged
mysys/my_open.c:
Auto merged
sql/handler.cc:
Auto merged
sql/handler.h:
Auto merged
sql/lex.h:
Auto merged
sql/sql_show.cc:
Auto merged
sql/sql_yacc.yy:
Auto merged
sql/table.cc:
Auto merged
sql/table.h:
Auto merged
storage/maria/ha_maria.cc:
Auto merged
storage/maria/ha_maria.h:
Auto merged
storage/maria/ma_bitmap.c:
Auto merged
storage/maria/ma_blockrec.h:
Auto merged
storage/maria/ma_create.c:
Auto merged
storage/maria/ma_dynrec.c:
Auto merged
storage/maria/ma_loghandler.h:
Auto merged
storage/maria/ma_loghandler_lsn.h:
Auto merged
storage/maria/ma_open.c:
Auto merged
storage/maria/ma_pagecache.c:
Auto merged
storage/maria/ma_pagecache.h:
Auto merged
storage/maria/ma_test1.c:
Auto merged
storage/maria/ma_test2.c:
Auto merged
storage/maria/ma_update.c:
Auto merged
storage/maria/maria_chk.c:
Auto merged
storage/maria/unittest/ma_test_loghandler-t.c:
Auto merged
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
Auto merged
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
Auto merged
storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
Auto merged
storage/myisam/ft_stopwords.c:
Auto merged
storage/myisam/mi_close.c:
Auto merged
support-files/compiler_warnings.supp:
Auto merged
mysys/mf_keycache.c:
Pulled from mysql-maria tree, manual merge.
mysys/my_handler.c:
Pulled from mysql-maria tree, manual merge.
mysys/my_symlink2.c:
Pulled from mysql-maria tree, manual merge.
sql/mysqld.cc:
Pulled from mysql-maria tree, manual merge.
sql/sql_table.cc:
Pulled from mysql-maria tree, manual merge.
storage/maria/ma_blockrec.c:
Pulled from mysql-maria tree, manual merge.
storage/maria/ma_check.c:
Pulled from mysql-maria tree, manual merge.
storage/maria/ma_close.c:
Pulled from mysql-maria tree, manual merge.
storage/maria/ma_loghandler.c:
Pulled from mysql-maria tree, manual merge.
storage/maria/maria_def.h:
Pulled from mysql-maria tree, manual merge.
into mysql.com:/home/my/mysql-maria
include/maria.h:
Auto merged
sql/handler.h:
Auto merged
storage/maria/ha_maria.cc:
Auto merged
storage/maria/ma_close.c:
Auto merged
storage/maria/ma_loghandler.c:
Auto merged
storage/maria/ma_open.c:
Auto merged
storage/maria/ma_pagecache.c:
Auto merged
storage/maria/ma_pagecache.h:
Auto merged
storage/maria/maria_chk.c:
Auto merged
storage/maria/ma_blockrec.c:
Manual merge (No changes)
storage/maria/ma_check.c:
Manual merge
storage/maria/ma_create.c:
Manual merge
storage/maria/ma_delete_all.c:
Manual merge
storage/maria/ma_init.c:
Manual merge (no changes)
storage/maria/ma_test_all.sh:
Manual merge
storage/maria/maria_def.h:
Manual merge
- WL#3240 "log DROP TABLE in Maria"
- similarly, log RENAME TABLE, REPAIR/OPTIMIZE TABLE, and
DELETE no_WHERE_clause (== the DELETE which just truncates the files)
- create_rename_lsn added to MARIA_SHARE's state
- all these operations (except DROP TABLE) also update the table's
create_rename_lsn, which is needed for the correctness of
Recovery (see function comment of _ma_repair_write_log_record()
in ma_check.c)
- write a COMMIT record when transaction commits.
- don't log REDOs/UNDOs if this is an internal temporary table
like inside ALTER TABLE (I expect this to be a big win). There was
already no logging for user-created "CREATE TEMPORARY" tables.
- don't fsync files/directories if the table is not transactional
- in translog_write_record(), autogenerate a 2-byte-id for the table
and log the "id->name" pair (LOGREC_FILE_ID); log
LOGREC_LONG_TRANSACTION_ID; automatically store
the table's 2-byte-id in any log record.
- preparations for Checkpoint: translog_get_horizon(); pausing Checkpoint
when some dirty pages are unknown; capturing trn->rec_lsn,
trn->first_undo_lsn for Checkpoint and log's low-water-mark computing.
- assertions, comments.
storage/maria/Makefile.am:
more files to build
storage/maria/ha_maria.cc:
- logging a REPAIR log record if REPAIR/OPTIMIZE was successful.
- ha_maria::data_file_type does not have to be set in every info()
call, just do it once in open().
- if caller said that transactionality can be disabled (like if
caller is ALTER TABLE) i.e. thd->transaction.on==FALSE, then we
temporarily disable transactionality of the table in external_lock();
that will ensure that no REDOs/UNDOs are logged for this possibly
massive write operation (they are not needed, as if any write fails,
the table will be dropped). We re-enable in external_lock(F_UNLCK),
which in ALTER TABLE happens before the tmp table replaces the original
one (which is good, as thus the final table will have a REDO RENAME
and a correct create_rename_lsn).
- when we commit we also have to write a log record, so
trnman_commit_trn() calls become ma_commit() calls
- at end of engine's initialization, we are potentially entering a
multi-threaded dangerous world (clients are going to be accepted)
and so some assertions of mutex-owning become enforceable, for that
we set maria_multi_threaded=TRUE (see ma_control_file.c)
storage/maria/ha_maria.h:
new member ha_maria::save_transactional (see also ha_maria.cc)
storage/maria/ma_blockrec.c:
- fixing comments according to discussion with Monty
- if a table is transactional but temporarily non-transactional
(like in ALTER TABLE), we need to give a sensible LSN to the pages
(and, if we give 0, pagecache asserts).
- translog_write_record() now takes care of storing the share's
2-byte-id in the log record
storage/maria/ma_blockrec.h:
fixing comment according to discussion with Monty
storage/maria/ma_check.c:
When REPAIR/OPTIMIZE modify the data/index file, if this is a
transactional table, they must sync it; if they remove files or rename
files, they must sync the directory, so that everything is durable.
This is just applying to REPAIR/OPTIMIZE the logic already implemented
in CREATE/DROP/RENAME a few months ago.
Adding a function to write a LOGREC_REPAIR_TABLE at end of
REPAIR/OPTIMIZE (called only by ha_maria, not by maria_chk), and
to update the table's create_rename_lsn.
storage/maria/ma_close.c:
fix for a future bug
storage/maria/ma_control_file.c:
ensuring that if Maria is running in multi-threaded mode, anybody
wanting to write to the control file and update
last_checkpoint_lsn/last_logno owns the log's lock.
storage/maria/ma_control_file.h:
see ma_control_file.c
storage/maria/ma_create.c:
when creating a table:
- sync it and its directory only if this is a transactional table
and there is a log (no point in syncing in maria_chk)
- decouple the two uses of linkname/linkname_ptr (for index file and
for data file) into more variables, as we need to know all links
until the moment we write the LOGREC_CREATE_TABLE.
- set share.data_file_type early so that _ma_initialize_data_file()
knows it (Monty's bugfix so that a table always has at least a bitmap
page when it is created; so data-file is not 0 bytes anymore).
- log a LOGREC_CREATE_TABLE; it contains the bytes which we have
just written to the index file's header. Update table's
create_rename_lsn.
- syncing of kfile had been bugified in a previous merge, correcting
- syncing of dfile is now needed as it's not empty anymore
- in _ma_initialize_data_file(), use share's block_size and not the
global one. This is a gratuitous change, both variables are equal,
just that I find it more future-proof to use share-bound variable
rather than global one.
storage/maria/ma_delete_all.c:
log a LOGREC_DELETE_ALL record when doing ma_delete_all_rows();
update create_rename_lsn then.
storage/maria/ma_delete_table.c:
- logging LOGREC_DROP_TABLE; knowing if this is needed, requires
knowing if the table is transactional, which requires opening the
table.
- we need to sync directories only if the table is transactional
storage/maria/ma_extra.c:
questions
storage/maria/ma_init.c:
when maria_end() is called, engine is not multithreaded
storage/maria/ma_loghandler.c:
- translog_inited has to be visible to ma_create() (see how it is used
in ma_create())
- checkpoint record will be a single record, not three
- no REDO for TRUNCATE (TRUNCATE calls ma_create() internally so will
log a REDO_CREATE)
- adding REDO for DELETE no_WHERE_clause (fast DELETE of all rows by
truncating the files), REPAIR.
- MY_WAIT_IF_FULL to wait&retry if a log write hits a full disk
- in translog_write_record(), if MARIA_SHARE does not yet have a
2-byte-id, generate one for it and log LOGREC_FILE_ID; automatically
store this short id into log records.
- in translog_write_record(), if transaction has not logged its
long trid, log LOGREC_LONG_TRANSACTION_ID.
- For Checkpoint, we need to know the current end-of-log: adding
translog_get_horizon().
- For Control File, adding an assertion that the thread owns the
log's lock (control file is protected by this lock)
storage/maria/ma_loghandler.h:
Changes in log records (see ma_loghandler.c).
new prototypes, new functions.
storage/maria/ma_loghandler_lsn.h:
adding a type LSN_WITH_FLAGS especially for TRN::first_undo_lsn,
where the most significant byte is used for flags.
storage/maria/ma_open.c:
storing the create_rename_lsn in the index file's header (in the
state, precisely) and retrieving it from there.
storage/maria/ma_pagecache.c:
- my set_if_bigger was wrong, correcting it
- if the first_in_switch list is not empty, it means that
changed_blocks misses some dirty pages, so Checkpoint cannot run and
needs to wait. A variable missing_blocks_in_changed_list is added to
tell that (should it be named missing_blocks_in_changed_blocks?)
- pagecache_collect_changed_blocks_with_lsn() now also tells the
minimum rec_lsn (needed for low-water mark computation).
storage/maria/ma_pagecache.h:
see ma_pagecache.c
storage/maria/ma_panic.c:
comment
storage/maria/ma_range.c:
comment
storage/maria/ma_rename.c:
- logging LOGREC_RENAME_TABLE; knowing if this is needed, requires
knowing if the table is transactional, which requires opening the
table.
- update create_rename_lsn
- we need to sync directories only if the table is transactional
storage/maria/ma_static.c:
comment
storage/maria/ma_test_all.sh:
- tip for Valgrind-ing ma_test_all
- do "export maria_path=somepath" before calling ma_test_all,
if you want to run ma_test_all out of storage/maria (useful
to have parallel runs, like one normal and one Valgrind, they
must not use the same tables so need to run in different directories)
storage/maria/maria_def.h:
- state now contains, in memory and on disk, the create_rename_lsn
- share now contains a 2-byte-id
storage/maria/trnman.c:
preparations for Checkpoint: capture trn->rec_lsn, trn->first_undo_lsn;
minimum first_undo_lsn needed to know log's low-water-mark
storage/maria/trnman.h:
using most significant byte of first_undo_lsn to hold miscellaneous
flags, for now TRANSACTION_LOGGED_LONG_ID.
dummy_transaction_object is already declared in ma_static.c.
storage/maria/trnman_public.h:
dummy_transaction_object was declared in all files including
trnman_public.h, while in fact it's a single object.
new prototype
storage/maria/unittest/ma_test_loghandler-t.c:
update for new prototype
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
update for new prototype
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
update for new prototype
storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
update for new prototype
storage/maria/ma_commit.c:
function which wraps:
- writing a LOGREC_COMMIT record (==commit on disk)
- calling trnman_commit_trn() (=commit in memory)
storage/maria/ma_commit.h:
new header file
.tree-is-private:
this file is now needed to keep our tree private (don't push it
to public trees). When 5.1 is merged into mysql-maria, we can abandon
our maria-specific post-commit trigger; .tree_is_private will take
care of keeping commit mails private. Don't push this file to public
trees.
specific
- adding TRN::first_undo_lsn, needed to know when a log can be deleted;
this variable must be set under log's mutex and that leads to setting
TRN::rec_lsn, TRN::undo_lsn and TRN::first_undo_lsn in a
inwrite_rec_hook; adding implementation of one hook for REDOs and one
for UNDOs. Thus translog_write_record() always uses TRN and so does
not need a short_id argument, can find it from TRN.
- Monty's patch for the last Valgrind error in the tree.
- Log handler's unit tests fail but Sanja says it's known
include/Makefile.am:
pagecache.h moved and renamed
include/maria.h:
pagecache.h moved and renamed
sql/handler.h:
pagecache.h moved and renamed
storage/maria/Makefile.am:
pagecache.h moved and renamed
storage/maria/ha_maria.cc:
adding an assertion which sounds logical
storage/maria/ma_blockrec.c:
trn->rec_lsn and trn->undo_lsn are now set via hooks inside the log
record's writing; this allows to also set trn->first_undo_lsn
needed to compute the log's low-water mark.
The PAGERANGE_STORE_SIZE -> PAGE_STORE_SIZE is Monty's fix to a
Valgrind error.
storage/maria/ma_loghandler.c:
"tcb" renamed to "trn". Log handler now knows what is a transaction,
and finds short_id from trn. trn's rec_lsn, undo_lsn, first_undo_lsn
are now set by some inwrite_rec_hookS (one for REDOs, one for UNDOs).
The HAVE_purify blocks are Monty's fix to a Valgrind error.
storage/maria/ma_loghandler.h:
Log handler functions use TRN, that needs a forward declaration
storage/maria/ma_pagecache.c:
pagecache.h was moved and renamed
storage/maria/ma_pagecache.h:
pagecache.h was moved and renamed
storage/maria/ma_pagecaches.c:
pagecache.h was moved and renamed
storage/maria/trnman.c:
initializing some members of TRN.
storage/maria/trnman.h:
TRN::first_undo_lsn needed for log's low-water mark calculation
(which will serve to know which logs can be deleted)
storage/maria/unittest/ma_test_loghandler-t.c:
translog_write_record() now needs a valid TRN
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
translog_write_record() now needs a valid TRN
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
translog_write_record() now needs a valid TRN
storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
translog_write_record() now needs a valid TRN
storage/maria/unittest/test_file.h:
pagecache.h was moved and renamed
