moved debugging defines.
Fixed buffer flushing (we copied last page
content before it was actually written, now
we abvance pointer in new buffer and unlock
it while waiting for filling old buffer)
Misc changes:
- fix for benign Valgrind error, compiler warnings
- fix for a segfault in execution of maria_delete_all_rows() and one
when taking multiple checkpoints
- fix for too paranoid assertion
- adding ability to take checkpoints at the end of the REDO phase
and at the end of recovery.
- other minor changes
storage/maria/ha_maria.cc:
The checkpoint done after Recovery is finished, is moved to
maria_recover().
storage/maria/ma_bitmap.c:
fix for Valgrind error: the "shadow debug copy" of the bitmap page
started unitialized and so ma_print_bitmap() would use it uninitialized
storage/maria/ma_checkpoint.c:
* reset pointers to NULL after freeing them, or we segfault at
next checkpoint in my_realloc().
* fix for compiler warnings.
storage/maria/ma_delete_all.c:
info->trn is NULL for non-transactional tables
storage/maria/ma_locking.c:
correct assertion (it fired wrongly in execution of REDO_DROP_TABLE
due to the maria_extra(HA_PREPARE_FOR_DROP)->_ma_decrement_open_count()
->maria_lock_database(F_UNLCK); another solution would have been to
not call _ma_decrement_open_count() (it's ok to have a wrong open
count in a table which we are dropping), but the same problem
would still exist for REDO_RENAME_TABLE.
storage/maria/ma_loghandler.c:
fail early if UNRECOVERABLE_ERROR
storage/maria/ma_recovery.c:
* new argument to maria_apply_log(): should it take checkpoints
(at end of REDO phase and at the very end) or no.
* moving the call to translog_next_LSN() into
parse_checkpoint_record() ("hide the details").
* Refining an error detection for something which could happen
if there is a checkpoint record in the log.
* Using close_one_table() instead of maria_extra(HA_EXTRA_PREPARE_FOR_DROP|RENAME),
as it looks safer, and also changing how close_one_table() works:
it now limits itself to scanning all_tables[], thus having one loopp
instead of two, which should be faster (as a result, it does not
close tables not registered in this array, which is ok as there
should not be any).
storage/maria/ma_recovery.h:
new parameter
storage/maria/maria_read_log.c:
update to new prototype
Debug output information fixed.
Fixed direct page referencing in write mode.
storage/maria/ma_loghandler.c:
TODO added.
storage/maria/ma_pagecache.c:
Debug output information fixed.
Fixed direct page referencing in write mode.
note: this does *not* move pagecache* variables.
sql/mysqld.cc:
remove maria_* variables from the server, declare them in ha_maria.cc
sql/set_var.cc:
remove maria_* variables from the server, declare them in ha_maria.cc
sql/sql_class.h:
remove maria_* variables from the server, declare them in ha_maria.cc
sql/sql_plugin.cc:
bugfix: support for ENUM/SET thd local plugin variables
storage/maria/ha_maria.cc:
remove maria_* variables from the server, declare them in ha_maria.cc
into a88-113-38-195.elisa-laajakaista.fi:/home/my/bk/mysql-maria.prod
BUILD/SETUP.sh:
Auto merged
sql/opt_range.cc:
Auto merged
sql/sql_insert.cc:
Auto merged
storage/maria/ha_maria.cc:
Auto merged
storage/maria/ma_check.c:
Auto merged
storage/maria/ma_create.c:
Auto merged
storage/maria/ma_delete.c:
Auto merged
storage/maria/ma_open.c:
Auto merged
storage/maria/ma_update.c:
Auto merged
storage/maria/ma_write.c:
Auto merged
storage/maria/maria_chk.c:
Auto merged
storage/maria/maria_def.h:
Auto merged
storage/maria/ma_pagecache.c:
Manual merge with maria tree.
storage/myisam/sort.c:
Manual merge with maria tree.
storage/maria/ma_test1.c:
After merge fixes. Applied changes made in myisam manually to maria.
Fixed memory leak.
storage/maria/ma_test2.c:
After merge fixes. Applied changes made in myisam manually to maria.
Fixed memory leak.
* Thanks to Serg's tip, we fix here the compilation issue of
REDO_REPAIR_TABLE's execution, by defining versions of
_ma_killed_ptr() and _ma_check_print_info|warning|error()
in maria_read_log.c (we move those of maria_chk.c into an include
file and include it in maria_chk.c and maria_read_log.c).
Execution of such record looks like working from my tests (it only
happens in maria_read_log; recovery-from-mysqld skips DDLs and
REPAIR is considered DDL here as it bypasses logging): tested
ALTER TABLE ENABLE KEYS and then remove table, apply log: that
did a repair.
* Recent changes broke maria_read_log a bit: -a became default
and -o caused error; fixing this.
storage/maria/Makefile.am:
addind new file
storage/maria/ma_recovery.c:
enable execution of REDO_REPAIR_TABLE by maria_read_log now that
it compiles. Now reason to keep only T_QUICK from testflag.
storage/maria/maria_chk.c:
moving these functions to ma_check_standalone.h for reusability
storage/maria/maria_def.h:
comment
storage/maria/maria_read_log.c:
ma_check_standalone.h needs my_progname_short.
Fixing bug where "maria_read_log" would default to -a and
"maria_read_log -o" would throw an error. Implemented behaviour is:
- no options: usage()
- -a : applys, applys UNDOs by default unless --disable-undo
- -o : only prints
storage/maria/ma_check_standalone.h:
All standalone programs which need to use functions from ma_check.c
(like maria_repair()) must define their version of _ma_killed_ptr()
and _ma_check_print_info|warning|error(). Indeed, linking with ma_check.o
brings in the dependencies of ma_check.o which are definitions of the above
functions; if the program does not define them then the ones of
ha_maria.o are used i.e. ha_maria.o is linked into the program, and this
brings dependencies of ha_maria.o on mysqld.o into the program's linking
which thus fails, as the program is not linked with mysqld.o.
We put in this file the functions which maria_chk.c uses, so that
they can be reused by maria_read_log (when it replays REDO_REPAIR_TABLE)
as they are good enough (they just print to stdout/stderr like
maria_read_log already does).
storage/maria/ma_pagecache.c:
pagecache->blocks is now long, takes 8 bytes on some platforms.
The cast to ulonglong in int8store is for those platforms where
ulong is 32-bit and int8store uses some << shifts, if there are
(x<<40 is undefined if x is 32-bit).
storage/maria/ma_recovery.c:
this change corresponds to the one done in ma_pagecache.c: number
of dirty pages is stored in 8 bytes.
* Recovery of the table's live checksum (CREATE TABLE ... CHECKSUM=1)
is achieved in this patch. The table's live checksum
(info->s->state.state.checksum) is updated in inwrite_rec_hook's
under the log mutex when writing UNDO_ROW_INSERT|UPDATE|DELETE
and REDO_DELETE_ALL. The checksum variation caused by the operation
is stored in these UNDOs, so that the REDO phase, when it sees such
UNDOs, can update the live checksum if it is older (state.is_of_lsn is
lower) than the record. It is also used, as a nice add-on with no
cost, to do less row checksum computation during the UNDO phase
(as we have it in the record already).
Doing this work, it became pressing to move in-write hooks
(write_hook_for_redo() et al) to ma_blockrec.c.
The 'parts' argument of inwrite_rec_hook is unpredictable (it comes
mangled at this stage, for example by LSN compression) so it is
replaced by a 'void* hook_arg', which is used to pass down information,
currently only to write_hook_for_clr_end() (previous undo_lsn and
type of undone record).
* If from ha_maria, we print to stderr how many seconds (with one
fractional digit) the REDO phase took, same for UNDO phase and for
final table close. Just to give an indication for debugging and maybe
also for Support.
storage/maria/ha_maria.cc:
question for Monty
storage/maria/ma_blockrec.c:
* log in-write hooks (write_hook_for_redo() etc) move from
ma_loghandler.c to here; this is natural: the hooks are coupled
to their callers (functions in ma_blockrec.c).
* translog_write_record() now has a new argument "hook_arg";
using it to pass down to write_hook_for_clr_end() the transaction's
previous_undo_lsn and the type of the being undone record, and also
to pass down to all UNDOs the live checksum variation caused by the
operation.
* If table has live checksum, store in UNDO_ROW_INSERT|UPDATE|DELETE
and in CLR_END the checksum variation ("delta") caused by the
operation. For example if a DELETE caused the table's live checksum
to change from 123 to 456, we store in the UNDO_ROW_DELETE, in 4 bytes,
the value 333 (456-123).
* Instead of hard-coded "1" as length of the place where we store
the undone record's type in CLR_END, use a symbol CLR_TYPE_STORE_SIZE;
use macros clr_type_store and clr_type_korr.
* write_block_record() has a new parameter 'old_record_checksum'
which is the pre-computed checksum of old_record; that value is used
to update the table's live checksum when writing UNDO_ROW_UPDATE|CLR_END.
* In allocate_write_block_record(), if we are executing UNDO_ROW_DELETE
the row's checksum is already computed.
* _ma_update_block_record2() now expect the new row's checksum into
cur_row.checksum (was already true) and the old row's checksum into
new_row.checksum (that's new). Its two callers, maria_update() and
_ma_apply_undo_row_update(), honour this.
* When executing an UNDO_ROW_INSERT|UPDATE|DELETE in UNDO phase, pick
up the checksum delta from the log record. It is then used to update
the table's live checksum when writing CLR_END, and saves us a
computation of record.
storage/maria/ma_blockrec.h:
in-write hooks move from ma_loghandler.c
storage/maria/ma_check.c:
more straightforward size of buffer
storage/maria/ma_checkpoint.c:
<= is enough
storage/maria/ma_commit.c:
new prototype of translog_write_record()
storage/maria/ma_create.c:
new prototype of translog_write_record()
storage/maria/ma_delete.c:
The row's checksum must be computed before calling(*delete_record)(),
not after, because it must be known inside _ma_delete_block_record()
(to update the table's live checksum when writing UNDO_ROW_DELETE).
If deleting from a transactional table, live checksum was already updated
when writing UNDO_ROW_DELETE.
storage/maria/ma_delete_all.c:
@todo is now done (in ma_loghandler.c)
storage/maria/ma_delete_table.c:
new prototype of translog_write_record()
storage/maria/ma_loghandler.c:
* in-write hooks move to ma_blockrec.c.
* translog_write_record() gets a new argument 'hook_arg' which is
passed down to pre|inwrite_rec_hook. It is more useful that 'parts'
for those hooks, because when those hooks are called, 'parts' has
possibly been mangled (like with LSN compression) and is so
unpredictable.
* fix for compiler warning (unused buffer_start when compiling without
debug support)
* Because checksum delta is stored into UNDO_ROW_INSERT|UPDATE|DELETE
and CLR_END, but only if the table has live checksum, these records
are not PSEUDOFIXEDLENGTH anymore, they are now VARIABLE_LENGTH (their
length is X if no live checksum and X+4 otherwise).
* add an inwrite_rec_hook for UNDO_ROW_UPDATE, which updates the
table's live checksum. Update it also in hooks of UNDO_ROW_INSERT|
DELETE and REDO_DELETE_ALL and CLR_END.
* Bugfix: when reading a record in translog_read_record(), it happened
that "length" became negative, because the function assumed that
the record extended beyond the page's end, whereas it may be shorter.
storage/maria/ma_loghandler.h:
* Instead of hard-coded "1" and "4", use symbols and macros
to store/retrieve the type of record which the CLR_END corresponds
to, and the checksum variation caused by the operation which logs the
record
* translog_write_record() gets a new argument 'hook_arg' which is
passed down to pre|inwrite_rec_hook. It is more useful that 'parts'
for those hooks, because when those hooks are called, 'parts' has
possibly been mangled (like with LSN compression) and is so
unpredictable.
storage/maria/ma_open.c:
fix for "empty body in if() statement" (when compiling without safemutex)
storage/maria/ma_pagecache.c:
<= is enough
storage/maria/ma_recovery.c:
* print the time that each recovery phase (REDO/UNDO/flush) took;
this is enabled only when recovering from ha_maria. Is it printed
n seconds with a fractional part of one digit (like 123.4 seconds).
* In the REDO phase, update the table's live checksum by using
the checksum delta stored in UNDO_ROW_INSERT|DELETE|UPDATE and CLR_END.
Update it too when seeing REDO_DELETE_ALL.
* In the UNDO phase, when executing UNDO_ROW_INSERT, if the table does
not have live checksum then reading the record's header (as done by
the master loop of run_undo_phase()) is enough; otherwise we
do a translog_read_record() to have the checksum delta ready
for _ma_apply_undo_row_insert().
* When at the end of the REDO phase we notice that there is an unfinished
group of REDOs, don't assert in debug binaries, as I verified that it
can happen in real life (with kill -9)
* removing ' in #error as it confuses gcc3
storage/maria/ma_rename.c:
new prototype of translog_write_record()
storage/maria/ma_test_recovery.expected:
Change in output of ma_test_recovery: now all live checksums of
original tables equal those of tables recreated by the REDO phase
and those of tables fixed by the UNDO phase. I.e. recovery of
the live checksum looks like working (which was after all the only
goal of this changeset).
I checked by hand that it's not just all live checksums which are
now 0 and that's why they match. They are the old values like
3757530372. maria.test has hard-coded checksum values in its result
file so checks this too.
storage/maria/ma_update.c:
* It's useless to put up HA_STATE_CHANGED in 'key_changed',
as we put up HA_STATE_CHANGED in info->update anyway.
* We need to compute the old and new rows' checksum before calling
(*update_record)(), as checksum delta must be known when logging
UNDO_ROW_UPDATE which is done by _ma_update_block_record(). Note that
some functions change the 'newrec' record (at least _ma_check_unique()
does) so we cannot move the checksum computation too early in the
function.
storage/maria/ma_write.c:
If inserting into a transactional table, live's checksum was
already updated when writing UNDO_ROW_INSERT. The multiplication
is a trick to save an if().
storage/maria/unittest/ma_test_loghandler-t.c:
new prototype of translog_write_record()
storage/maria/unittest/ma_test_loghandler_first_lsn-t.c:
new prototype of translog_write_record()
storage/maria/unittest/ma_test_loghandler_max_lsn-t.c:
new prototype of translog_write_record()
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
new prototype of translog_write_record()
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
new prototype of translog_write_record()
storage/maria/unittest/ma_test_loghandler_noflush-t.c:
new prototype of translog_write_record()
storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
new prototype of translog_write_record()
storage/maria/unittest/ma_test_loghandler_purge-t.c:
new prototype of translog_write_record()
storage/myisam/sort.c:
fix for compiler warnings in pushbuild (write_merge_key* functions
didn't have their declaration match MARIA_HA::write_key).
gprof build for amd64.
storage/maria/ma_loghandler.c:
fater way to detect last page address for the last log file.
BUILD/compile-amd64-gprof-no-ndb:
New BitKeeper file ``BUILD/compile-amd64-gprof-no-ndb''
BitKeeper/etc/ignore:
Added libmysql_r/client_settings.h libmysqld/scheduler.cc libmysqld/sql_connect.cc libmysqld/sql_tablespace.cc to the ignore list
REDO optimization (Bascily avoid moving blocks from/to pagecache)
More command line arguments to maria_read_log
Fixed recovery bug when recreating table
sql/opt_range.cc:
Remove SAFE_MODE for opt_range as it disables UPDATE to use keys
storage/maria/ma_blockrec.c:
REDO optimization
Use new interface for pagecache_reads to avoid copying page buffers
storage/maria/ma_loghandler.c:
Patch from Sanja:
- Added new parameter to translog_get_page to use direct links to pagecache
- Changed scanner to be able to use direct links
This avoids a lot of calls to bmove512() in page cache.
storage/maria/ma_loghandler.h:
Added direct link to pagecache objects
storage/maria/ma_open.c:
Added const to parameter
Added missing braces
storage/maria/ma_pagecache.c:
From Sanja:
- Added direct links to pagecache (from pagecache_read())
Dirrect link means that on pagecache_read we get back a pointer to the pagecache buffer
From Monty:
- Fixed arguments to init_page_cache to handle big page caches
- Fixed compiler warnings
- Replaced PAGECACHE_PAGE_LINK with PAGECACHE_BLOCK_LINK * to catch errors
storage/maria/ma_pagecache.h:
Changed block numbers from int to long to be able to handle big page caches
Changed some PAGECACHE_PAGE_LINK to PAGECACHE_BLOCK_LINK
storage/maria/ma_recovery.c:
Fixed recovery bug when recreating table (table was kept open)
Moved some variables to function start (portability)
Added space to some print messages
storage/maria/maria_chk.c:
key_buffer_size -> page_buffer_size
storage/maria/maria_def.h:
Changed default page_buffer_size to 10M
storage/maria/maria_read_log.c:
Added more startup options:
--version
--undo (apply undo)
--page_cache_size (to run with big cache sizes)
--silent (to not get any output from --apply)
storage/maria/unittest/ma_control_file-t.c:
Fixed compiler warning
storage/maria/unittest/ma_test_loghandler-t.c:
Added new argument to translog_init_scanner()
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
Added new argument to translog_init_scanner()
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
Added new argument to translog_init_scanner()
Minor change: decouple the writing of a recovery trace from
compilation with --with-debug; that writing happens if and only
if EXTRA_DEBUG. So one can enable (by defining EXTRA_DEBUG) a recovery trace
in a non-debug build.
storage/maria/ma_recovery.c:
Small optimization. Decouple recovery trace from --with-debug.
Progress reports on stderr if doing recovery from ha_maria;
don't do checkpoints if activity since last checkpoint < 2MB
(no change in fact as background thread is disabled for now);
recovery trace is only if EXTRA_DEBUG now (better for benchmarks).
storage/maria/ma_checkpoint.c:
don't do checkpoints if activity (log writes plus page flushes)
since last checkpoint was < 2MB.
storage/maria/ma_recovery.c:
progress reports in recovery (10%, transactions left to rollback etc);
that is only if from ha_maria and is displayed on stderr.
Recovery trace is now created only if EXTRA_DEBUG.
storage/maria/ma_test_recovery.expected:
update (--debug gone)
storage/maria/ma_test_recovery:
don't use --debug, as it can absent from binary
record (for example in ma_test_loghandler-t). Reason was
wrongly matched () in DBUG.
storage/maria/ma_loghandler.c:
Wrongly matched parenthesis:
DBUG_PRINT(keyword, argslist) expands to roughly
_db_doprnt arglist;
So DBUG_PRINT("enter",(a);
b;
c);
expands to roughly
_db_doprnt(a);b;c;
which is valid code. Except that in non-debug builds, DBUG_PRINT(
expands to nothing so the wrongly "included" code is thrown away,
leading to some members of "scanner" to not be initialized.
Note that non-debug build fails in log handler functions, mail sent.
storage/maria/ma_blockrec.c:
fix for compiler warning
storage/maria/ma_checkpoint.c:
Debug build does not catch this situation
static int f();
...
f(2);
...
static int f(int a, int b);
Maybe this is because it believes the declaration is K&R. Non-debug
build catches it. Adding (void) as an habit to avoid such errors.
storage/maria/ma_checkpoint.h:
adding (void)
storage/maria/ma_recovery.c:
adding (void)
storage/maria/ma_recovery.h:
adding (void)
BLOB whose size didn't change. Fix for probably impossible problem
in Recovery.
mysql-test/r/maria.result:
result for new test
mysql-test/t/maria.test:
testcase for a bug (used to segfault)
storage/maria/ma_blockrec.c:
When writing a record, we put BLOBs into the head part if there is
room for them. "Is there room" was first decided by
!(tmp_data + length > end_of_data) (line 1894)
but then was tested again as
*blob_lengths < (ulong)(end_of_data - data). We see that in case of
equality, the first condition was true but the second was not,
so it was inconsistent and crashed later.
storage/maria/ma_recovery.c:
When wondering if recovery should update the state (like state.records):
if table was closed, its is_of_horizon was set to X, then
table was reopened and a REDO was written. If this REDO had LSN
X (as horizon is just a lower bound of the LSN of the next record),
we have to apply it. In practice this equality probably could not
happen because of LOGREC_FILE_ID would be written before the REDO.
* recovery from ha_maria now skips replaying DDLs (too dangerous)
* maria_read_log still replays DDLs, print warning about issues
* fixes to replaying of REDO_RENAME
* don't replay DDLs on corrupted tables (safer)
* print a one-line message when really doing a recovery (applies to
ha_maria, not maria_read_log) i.e. some REDOs or UNDOs are read.
storage/maria/ma_checkpoint.c:
fix for assertion failure
storage/maria/ma_recovery.c:
* Recovery from ha_maria now skips replaying DDLs (as the initial
plan said) as this is unsafe in case of crashes during the DDL;
applying the records may do harm (destroy important files)
so we prefer to leave the "mess" of files untouched. A proper
recovery of DDLs requires very careful thinking, probably testing
separately the existence of the data and index file instead of
using maria_open() which tests the existence of both, and maybe
storing create_rename_lsn in the data file too.
* maria_read_log still replays DDLs, we print a warning about dangers
(due to ALTER TABLE not logging insertions into the tmp table; we
will maybe need an option to have logging of those insertions).
* fixes to replaying of REDO_RENAME (test create_rename_lsn of 'new_name'
table if it exists; if that table exists and is more recent than the
record, remove the 'old_name' table).
* don't replay DDLs on corrupted tables (play safe)
* fail also in non-debug builds if table is open when it should not be
(when creating it for example, it should not be already open).
* when the trace file is not stdout (i.e. when this is ha_maria),
if really doing a recovery (reading REDOs or UNDOs), print a one-line
message to stderr to inform about start and end of recovery
(useful to know what mysqld is doing, especially if it takes long
or crashes).
storage/maria/ma_recovery.h:
parameter to replay DDLs or not
storage/maria/maria_read_log.c:
replay DDLs in maria_read_log, to be able to recreate tables from
scratch.
horizon addres.
storage/maria/ma_loghandler.c:
New function to get correct LSN from chunk address.
storage/maria/ma_loghandler.h:
New function to get correct LSN from chunk address.
storage/maria/ma_checkpoint.c:
The new macro for easier printing LSN added.
storage/maria/ma_loghandler.c:
The assertion returned.
The new macro for easier printing LSN added.
storage/maria/ma_loghandler_lsn.h:
The new macro for easier printing LSN added.
storage/maria/ma_pagecache.c:
The new macro for easier printing LSN added.
storage/maria/ma_recovery.c:
Recovery checks empty log state.
RECHEADER_READ_ERROR means some real error.
storage/maria/maria_read_log.c:
Read log starts from real beggining of the log and precess
error and empty log states.
The new macro for easier printing LSN added.
storage/maria/unittest/ma_test_loghandler-t.c:
The new macro for easier printing LSN added.
storage/maria/unittest/ma_test_loghandler_first_lsn-t.c:
The new macro for easier printing LSN added.
storage/maria/unittest/ma_test_loghandler_max_lsn-t.c:
The new macro for easier printing LSN added.
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
The new macro for easier printing LSN added.
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
The new macro for easier printing LSN added.
storage/maria/unittest/ma_test_loghandler_noflush-t.c:
The new macro for easier printing LSN added.
failed => Maria didn't start => tables were created as MyISAM).
storage/maria/ma_checkpoint.c:
safemalloc complains if my_realloc() is passed NULL and
MY_ALLOW_ZERO_PTR is not used.
* added replaying of REDO_REPAIR_TABLE, but disabled it as
mysterious linker errors appear.
* after replaying RENAME/REPAIR, we must bump create_rename_lsn
for idempotency of maria_read_log.
sql/mysqld.cc:
typo
storage/maria/ma_checkpoint.c:
silence compiler warning
storage/maria/ma_recovery.c:
* added replaying of REDO_REPAIR_TABLE, but disabled it as
mysterious linker errors appear.
* after replaying RENAME/REPAIR, we must bump create_rename_lsn
for idempotency of maria_read_log.
Finally this is the real checkpoint code.
It however exhibits unstabilities when a checkpoint runs concurrently
with data-modifying clients (table corruption, transaction log's
assertions) so for now a checkpoint is taken only at startup after
recovery and at shutdown, i.e. not in concurrent situations. Later
we will let it run periodically, as well as flush dirty pages
periodically (almost all needed code is there already, only pagecache
code is written but not committed).
WL#3072 Maria recovery
* replacing UNDO_ROW_PURGE with CLR_END; testing of those CLR_END via
ma_test2 which has INSERTs failing with duplicate keys.
* replaying of REDO_RENAME_TABLE
Now, off to test Recovery in ha_maria :)
BitKeeper/deleted/.del-ma_least_recently_dirtied.c:
Delete: storage/maria/ma_least_recently_dirtied.c
BitKeeper/deleted/.del-ma_least_recently_dirtied.h:
Delete: storage/maria/ma_least_recently_dirtied.h
storage/maria/Makefile.am:
compile Checkpoint module
storage/maria/ha_maria.cc:
When ha_maria starts, do a recovery from last checkpoint.
Take a checkpoint when that recovery has ended and when ha_maria
shuts down cleanly.
storage/maria/ma_blockrec.c:
* even if my_sync() fails we have to my_close() (otherwise we leak
a descriptor)
* UNDO_ROW_PURGE is replaced by a simple CLR_END for UNDO_ROW_INSERT,
as promised in the old comment; it gives us skipping during the
UNDO phase.
storage/maria/ma_check.c:
All REDOs before create_rename_lsn are ignored by Recovery. So
create_rename_lsn must be set only after all data/index has been
flushed and forced to disk. We thus move write_log_record_for_repair()
to after _ma_flush_tables_files_after_repair().
storage/maria/ma_checkpoint.c:
Checkpoint module.
storage/maria/ma_checkpoint.h:
optional argument if caller wants a thread to periodically take
checkpoints and flush dirty pages.
storage/maria/ma_create.c:
* no need to init some vars as the initial bzero(share) takes care of this.
* update to new function's name
* even if we fail in my_sync() we have to my_close()
storage/maria/ma_extra.c:
Checkpoint reads share->last_version under intern_lock, so we make
maria_extra() update it under intern_lock. THR_LOCK_maria still needed
because of _ma_test_if_reopen().
storage/maria/ma_init.c:
destroy checkpoint module when Maria shuts down.
storage/maria/ma_loghandler.c:
* UNDO_ROW_PURGE gone (see ma_blockrec.c)
* we need to remember the LSN of the LOGREC_FILE_ID for a share,
because this LSN is needed into the checkpoint record (Recovery wants
to know the validity domain of an id->name mapping)
* translog_get_horizon_no_lock() needed for Checkpoint
* comment about failing assertion (Sanja knows)
* translog_init_reader_data() thought that translog_read_record_header_scan()
returns 0 in case of error, but 0 just means "0-length header".
* translog_assign_id_to_share() now needs the MARIA_HA because
LOGREC_FILE_ID uses a log-write hook.
* Verify that (de)assignment of share->id happens only under intern_lock,
as Checkpoint reads this id with intern_lock.
* translog_purge() can accept TRANSLOG_ADDRESS, not necessarily
a real LSN.
storage/maria/ma_loghandler.h:
prototype updates
storage/maria/ma_open.c:
no need to initialize "res"
storage/maria/ma_pagecache.c:
When taking a checkpoint, we don't need to know the maximum rec_lsn
of dirty pages; this LSN was intended to be used in the two-checkpoint
rule, but last_checkpoint_lsn is as good.
4 bytes for stored_list_size is enough as PAGECACHE::blocks (number
of blocks which the pagecache can contain) is int.
storage/maria/ma_pagecache.h:
new prototype
storage/maria/ma_recovery.c:
* added replaying of REDO_RENAME_TABLE
* UNDO_ROW_PURGE gone (see ma_blockrec.c), replaced by CLR_END
* Recovery from the last checkpoint record now possible
* In new_table() we skip the table if the id->name mapping is older than
create_rename_lsn (mapping dates from lsn_of_file_id).
* in get_MARIA_HA_from_REDO_record() we skip the record
if the id->name mapping is newer than the record (can happen if processing
a record which is before the checkpoint record).
* parse_checkpoint_record() has to return a LSN, that's what caller expects
storage/maria/ma_rename.c:
new function's name; log end zeroes of tables' names (ease recovery)
storage/maria/ma_test2.c:
* equivalent of ma_test1's --test-undo added (named -u here).
* -t=1 now stops right after creating the table, so that
we can test undoing of INSERTs with duplicate keys (which tests the
CLR_END logged by _ma_write_abort_block_record()).
storage/maria/ma_test_recovery.expected:
Result of testing undoing of INSERTs with duplicate keys; there are
some differences in maria_chk -dvv but they are normal (removing
records does not shrink data/index file, does not put back the
"analyzed, optimized keys"(etc) index state.
storage/maria/ma_test_recovery:
Test undoing of INSERTs with duplicate keys, using ma_test2;
when such INSERT happens, it logs REDO_INSERT, UNDO_INSERT, REDO_DELETE,
CLR_END; we abort after that, and test that CLR_END causes recovery
to jump over UNDO_INSERT.
storage/maria/ma_write.c:
comment
storage/maria/maria_chk.c:
comment
storage/maria/maria_def.h:
* a new bit in MARIA_SHARE::in_checkpoint, used to build a list
of unique shares during Checkpoint.
* MARIA_SHARE::lsn_of_file_id added: the LSN of the last LOGREC_FILE_ID
for this share; needed to know to which LSN domain the mappings
found in the Checkpoint record apply (new mappings should not apply
to old REDOs).
storage/maria/trnman.c:
* small changes to how trnman_collect_transactions() fills its buffer;
it also uses a non-dummy lsn_read_non_atomic() found in ma_checkpoint.h
* testing of execution of UNDO_ROW_UPDATE
* when executing an UNDO_ROW_UPDATE, store "UNDO_ROW_UPDATE" as
"type of undone record" into the CLR_END record.
storage/maria/ma_blockrec.c:
When logging a CLR_END in write_block_record(), it can be for
a DELETE or for an UPDATE (now that Monty has coded execution of
UNDO_UPDATE)
storage/maria/ma_loghandler.c:
UNDO_ROW_UPDATE's execution coded, so no crash
storage/maria/ma_recovery.c:
UNDO_ROW_UPDATE's execution now coded, so no crash
storage/maria/ma_test1.c:
upper case letter
storage/maria/ma_test_recovery.expected:
output of testing execution of UNDO_ROW_UPDATE. Table's checksum
not recovered (known issue not specific to UPDATE).
storage/maria/ma_test_recovery:
Test execution of UNDO_ROW_UPDATE: first we stop ma_test1 after
deletes and commit, then we stop ma_test1 after updates and abort;
we verify that updates are rolled back by comparing tables
Fixed bug in undo_delete
Fixed wrong error output from maria_check
include/my_base.h:
Added marker if we have null fields in table
mysql-test/r/maria.result:
checksum in maria now ignore null fields that are null
sql/sql_table.cc:
Ignore null fields that are now
(Before enabling this, we have to change MyISAM to also skip null fields)
storage/maria/ma_blockrec.c:
More logging
After merge fixes
Fixed some bugs when using undo of VARCHAR fields
Fixed bug in undo_delete (We can't use info->rec_buff here as this is used in write_block_record())
storage/maria/ma_blockrec.h:
ma_recordpos_to_dir_entry changed to return uint
storage/maria/ma_check.c:
Fixed wrong output in case of errors
storage/maria/ma_create.c:
Set share.base.pack_reclength more correct for block record
Delete support for RAID
storage/maria/ma_open.c:
Don't calculate checksum fields with value NULL
storage/maria/ma_test1.c:
Fixed output from -v for VARCHAR keys
storage/maria/ma_test_recovery.expected:
Update results after adding new printf
New checksums (because we now ignore nulls)
Some file lengths are different, but think they are ok (didn't have time to investigate)
storage/myisam/ha_myisam.cc:
Fixed comment
storage/myisam/mi_test1.c:
Fixed bug
into mysql.com:/home/my/mysql-maria
storage/maria/ma_check.c:
Auto merged
storage/maria/ma_locking.c:
Auto merged
storage/maria/ma_loghandler.c:
Auto merged
storage/maria/ma_open.c:
Auto merged
storage/maria/ma_recovery.c:
Auto merged
storage/maria/maria_def.h:
Auto merged
storage/maria/maria_read_log.c:
Auto merged
storage/maria/ma_blockrec.c:
Manual merge
storage/maria/ma_test1.c:
Manual merge (using Guilhems code)
Fixed bug in duplicate key handling for block records during repair
All read-row methods now return error number in case of error
Don't calculate checksum for null fields
Fixed bug when running maria_read_log with -o
BUILD/SETUP.sh:
Added STACK_DIRECTION
BUILD/compile-pentium-debug-max:
Moved STACK_DIRECTION to SETUP
include/myisam.h:
Added extra parameter to write_key
storage/maria/ma_blockrec.c:
Added applying of undo for updates
Fixed indentation
Removed some not needed casts
Fixed wrong logging of CLR record
Split ma_update_block_record to two functions to be able to reuse it from undo-applying
Simplify filling of packed fields
ma_record_block_record) now returns error number on failure
Sligtly changed log record information for undo-update
storage/maria/ma_check.c:
Fixed bug in duplicate key handling for block records during repair
storage/maria/ma_checksum.c:
Don't calculate checksum for null fields
storage/maria/ma_dynrec.c:
_ma_read_dynamic_reocrd() now returns error number on error
Rest of the changes are code simplification and indentation fixes
storage/maria/ma_locking.c:
Added comment
storage/maria/ma_loghandler.c:
More debugging
Removed printing of total_record_length as this was always same as record_length
storage/maria/ma_open.c:
Allocate bitmap for changed fields
storage/maria/ma_packrec.c:
read_record now returns error number on error
storage/maria/ma_recovery.c:
Fixed wrong arguments to undo_row_update
storage/maria/ma_statrec.c:
read_record now returns error number on error (not 1)
Code simplification
storage/maria/ma_test1.c:
Added exit possibility after update phase (to test undo of updates)
storage/maria/maria_def.h:
Include bitmap header file
storage/maria/maria_read_log.c:
Fixed bug when running with -o
end of translog_flush() when datadir was in /dev/shm.
storage/maria/ma_loghandler.c:
directory syncing can fail on shared memory devices (/dev/shm on Linux
in this case); see my_sync_dir().
for compiler errors in pushbuild). Small bugfix.
sql/handler.h:
don't use pagecaches if no Maria
storage/maria/ma_check.c:
correcting mistake in previous push; need to call this function
otherwise create_rename_lsn would not be updated at end of REPAIR.
