Essentially, the problem is that safemalloc is excruciatingly
slow as it checks all allocated blocks for overrun at each
memory management primitive, yielding a almost exponential
slowdown for the memory management functions (malloc, realloc,
free). The overrun check basically consists of verifying some
bytes of a block for certain magic keys, which catches some
simple forms of overrun. Another minor problem is violation
of aliasing rules and that its own internal list of blocks
is prone to corruption.
Another issue with safemalloc is rather the maintenance cost
as the tool has a significant impact on the server code.
Given the magnitude of memory debuggers available nowadays,
especially those that are provided with the platform malloc
implementation, maintenance of a in-house and largely obsolete
memory debugger becomes a burden that is not worth the effort
due to its slowness and lack of support for detecting more
common forms of heap corruption.
Since there are third-party tools that can provide the same
functionality at a lower or comparable performance cost, the
solution is to simply remove safemalloc. Third-party tools
can provide the same functionality at a lower or comparable
performance cost.
The removal of safemalloc also allows a simplification of the
malloc wrappers, removing quite a bit of kludge: redefinition
of my_malloc, my_free and the removal of the unused second
argument of my_free. Since free() always check whether the
supplied pointer is null, redudant checks are also removed.
Also, this patch adds unit testing for my_malloc and moves
my_realloc implementation into the same file as the other
memory allocation primitives.
client/mysqldump.c:
Pass my_free directly as its signature is compatible with the
callback type -- which wasn't the case for free_table_ent.
Interface for maria extensions.
Alternative plugin interface with additional info (maturity and string version).
CMakeLists.txt:
Maria plugin interface used.
config/ac-macros/plugins.m4:
Maria plugin interface used.
configure.in:
Maria plugin interface used.
include/mysql/plugin.h:
Maria plugin interface added.
include/mysql/plugin_auth.h.pp:
Maria plugin interface added.
plugin/auth/auth_socket.c:
Maria plugin interface added.
plugin/auth/dialog.c:
Maria plugin interface added.
plugin/daemon_example/daemon_example.cc:
Maria plugin interface added.
plugin/fulltext/plugin_example.c:
Maria plugin interface added.
sql/ha_ndbcluster.cc:
Maria plugin interface added.
sql/ha_partition.cc:
Maria plugin interface added.
sql/log.cc:
Maria plugin interface added.
sql/sql_acl.cc:
Maria plugin interface added.
sql/sql_builtin.cc.in:
Maria plugin interface used.
sql/sql_plugin.cc:
Maria plugin interface added.
sql/sql_plugin.h:
Maria plugin interface used.
sql/sql_show.cc:
Maria plugin interface added.
storage/archive/ha_archive.cc:
Maria plugin interface added.
storage/blackhole/ha_blackhole.cc:
Maria plugin interface added.
storage/csv/ha_tina.cc:
Maria plugin interface added.
storage/example/ha_example.cc:
Maria plugin interface added.
storage/federated/ha_federated.cc:
Maria plugin interface added.
storage/federatedx/ha_federatedx.cc:
Maria plugin interface added.
storage/heap/ha_heap.cc:
Maria plugin interface added.
storage/ibmdb2i/ha_ibmdb2i.cc:
Maria plugin interface added.
storage/innobase/handler/ha_innodb.cc:
Maria plugin interface added.
storage/innodb_plugin/handler/i_s.cc:
Maria plugin interface added.
storage/maria/ha_maria.cc:
Maria plugin interface added.
storage/myisam/ha_myisam.cc:
Maria plugin interface added.
storage/myisammrg/ha_myisammrg.cc:
Maria plugin interface added.
storage/pbxt/src/ha_pbxt.cc:
Maria plugin interface added.
storage/xtradb/handler/ha_innodb.cc:
Maria plugin interface added.
storage/xtradb/handler/i_s.cc:
Maria plugin interface added.
storage/xtradb/handler/i_s.h:
Maria plugin interface added.
This patch:
- Moves all definitions from the mysql_priv.h file into
header files for the component where the variable is
defined
- Creates header files if the component lacks one
- Eliminates all include directives from mysql_priv.h
- Eliminates all circular include cycles
- Rename time.cc to sql_time.cc
- Rename mysql_priv.h to sql_priv.h
There was no way to repair corrupt ARCHIVE data file,
when unrecoverable data loss is inevitable.
With this fix REPAIR ... EXTENDED attempts to restore
as much rows as possible, ignoring unrecoverable data.
Normal REPAIR is still able to repair meta-data file
only.
mysql-test/r/archive.result:
A test case for BUG#46565.
mysql-test/std_data/bug46565.ARZ:
A test case for BUG#46565.
mysql-test/std_data/bug46565.frm:
A test case for BUG#46565.
mysql-test/t/archive.test:
A test case for BUG#46565.
storage/archive/ha_archive.cc:
Allow unrecoverable data loss when extended repair
is requested.
Server crashes when accessing ARCHIVE table with missing
.ARZ file.
When opening a table, ARCHIVE didn't properly pass through
error code from lower level azopen() to higher level open()
method.
mysql-test/r/archive.result:
A test case for BUG#48757.
mysql-test/t/archive.test:
A test case for BUG#48757.
storage/archive/ha_archive.cc:
Pass through error code from azopen().
- Marked a couple of tests with --big
- Fixed xtradb/handler/ha_innodb.cc to call explain_filename()
storage/xtradb/handler/ha_innodb.cc:
Call explain_filename() to get proper names for partitioned tables
Select queries on archive tables when joined on their primary keys
returns no results(empty set)
Archive storage doesn't inform the handler about the fetched record
status when it is found. Fixed the archive storage engine to update
the record status when it fetches successfully
- Moved some code from innodb_plugin to xtradb, to ensure that all tests runs
- Did changes in pbxt and maria storage engines becasue of changes in thd->query
- Reverted wrong code in sql_table.cc for how ROW_FORMAT is used.
This is a re-commit of Monty's merge to eliminate an extra commit from
MySQL-5.1.42 that was accidentally included in the merge.
This is a merge of the MySQL 5.1.41 clone-off (clone-5.1.41-build). In
case there are any extra changes done before final MySQL 5.1.41
release, these will need to be merged later before MariaDB 5.1.41
release.
on any access
Archive engine for 5.1 (and latter) version uses a modified
version of zlib (azlib). These two version are incompatible
so a proper upgrade is needed before tables created in 5.0
can be used reliable.
This upgrade can be performed using repair. But due to lack
of test its risky to allow upgrade for now. This patch addresses
only the crashing issue. Any attempt to repair will be blocked.
Eventually repair can be allowed to run through (which will also
cause an upgrade from older version to newer) but only after a
thorough testing.
mysql-test/r/archive.result:
Updated result file for test case for bug#47012
mysql-test/std_data/bug47012.ARM:
part of archive table (t1) created in mysql 5.0
mysql-test/std_data/bug47012.ARZ:
part of archive table (t1) created in mysql 5.0
mysql-test/std_data/bug47012.frm:
part of archive table (t1) created in mysql 5.0
mysql-test/t/archive.test:
Added test case for bug#47012.
storage/archive/azio.c:
Fixed a minor issues (minor version overwriting version in
stream structure)
Removed assertion when an older version is found. Instead
setting the correct version (2) in s->version
If an unknown version is found marked it as corrupt.
storage/archive/ha_archive.cc:
Detecting the archive version in getShare and marking
it as need to upgrade.
Blocking open if the archive needs an upgrade. This
can be allowed in case of open for repair to upgrade
the archive but needs to tested.
Archive engine returns wrong values for average record length
and max data length.
With this fix they're calculated as following:
- max data length is 2 ^ 63 where large files are supported
and INT_MAX32 where this is not supported;
- average record length is data length / records in data file.
mysql-test/r/archive.result:
A test case for BUG#29203.
mysql-test/t/archive.test:
A test case for BUG#29203.
storage/archive/ha_archive.cc:
Better estimation for average row length and maximal data
file length.
* Finished Monty and Jani's merge
* Some InnoDB tests still fail (because it's old xtradb code run against
newer testsuite). They are expected to go after mergning with the latest
xtradb.
SELECT with join (not only self-join) from archive table may
return incomplete result set, when result set size exceeds
join buffer size.
The problem was that archive row counter was initialzed too
early, when ha_archive::info() method was called. Later,
when optimizer exceeds join buffer, it attempts to reuse
handler without calling ha_archive::info() again (which is
correct).
Fixed by moving row counter initialization from
ha_archive::info() to ha_archive::rnd_init().
mysql-test/r/archive.result:
A test case for BUG#46961.
mysql-test/t/archive.test:
A test case for BUG#46961.
storage/archive/ha_archive.cc:
Since a cursor may get reused without a call to ::info(),
move assignment of scan_rows to a proper place, that is
::rnd_init().
Problem was that ha_archive::info did not use the flag argument
correctly
fixed so that it updated the correct values depending on the flag.
mysql-test/r/partition_archive.result:
Bug#44622: Using PARTITIONs with ARCHIVE engine reports 0 bytes in i_s.TABLES
Updated testresult
mysql-test/t/partition_archive.test:
Bug#44622: Using PARTITIONs with ARCHIVE engine reports 0 bytes in i_s.TABLES
Added testcase
storage/archive/ha_archive.cc:
Bug#44622: Using PARTITIONs with ARCHIVE engine reports 0 bytes in i_s.TABLES
Fixed ha_archive::info() so that it updated the correct values
depending on the flag. (did it all only if HA_STATUS_TIME before...)
bzr branch mysql-5.1-performance-version mysql-trunk # Summit
cd mysql-trunk
bzr merge mysql-5.1-innodb_plugin # which is 5.1 + Innodb plugin
bzr rm innobase # remove the builtin
Next step: build, test fixes.
This patch fixes compilation warning, "conversion from 'time_t' to 'ulong',
possible loss of data".
The fix is to typecast time_t to ulong before assigning it to ulong.
Backported this from 6.0-bugteam tree.
storage/archive/ha_archive.cc:
type casting time_t to ulong before assigning.
storage/federated/ha_federated.cc:
type casting time_t to ulong before assigning.
storage/innobase/handler/ha_innodb.cc:
type casting time_t to ulong before assigning.
storage/myisam/ha_myisam.cc:
type casting time_t to ulong before assigning.
It leads to crash because there is no OOM check in ha_archive::unpack_row().
The fix:
added OOM error check
mysql-test/r/archive.result:
remover test case
mysql-test/std_data/bug32880.ARN:
remover test case
mysql-test/std_data/bug32880.ARZ:
remover test case
mysql-test/std_data/bug32880.frm:
remover test case
mysql-test/t/archive.test:
remover test case
Any statement reading corrupt archive data file
(CHECK/REPAIR/SELECT/UPDATE/DELETE) may cause assertion
failure in debug builds. This assertion has been removed
and an error is returned instead.
Also fixed that CHECK/REPAIR returns vague error message
when it mets corruption in archive data file. This is
fixed by returning proper error code.
mysql-test/r/archive.result:
A test case for BUG#32880
mysql-test/std_data/bug32880.ARN:
corrupted archive table to test check and repair table operation
mysql-test/std_data/bug32880.ARZ:
corrupted archive table to test check and repair table operation
mysql-test/std_data/bug32880.frm:
corrupted archive table to test check and repair table operation
mysql-test/t/archive.test:
A test case for BUG#32880
storage/archive/ha_archive.cc:
Fixed unpack_row() to return the error instead of throwing assertion
and also fixed repair() to throw better error when repair table
operation fails on corrupted archive table
case and then select
Problem was that the archive share was using a case insensitive
charset when comparing table names
Solution was to use a case sensitive char set when the table
names are case sensitive
mysql-test/suite/parts/r/partition_mgm_lc0_archive.result:
Bug#37719: Crash if rename Archive table to same name with different
case and then select
Updated to correct result.
storage/archive/ha_archive.cc:
Bug#37719: Crash if rename Archive table to same name with different
case and then select
system_charset_info is case insensitive, table_alias_charset depends
on the filesystem/lower_case_table_names variable.
since there could be two tables that used the same share, unpredicted
things could happen.
Added argument to maria_end_bulk_insert() to know if the table will be deleted after the operation
Fixed wrong call to strmake
Don't call bulk insert in case of inserting only one row (speed optimization as starting/stopping bulk insert
Allow storing year 2155 in year field
When running with purify/valgrind avoid copying structures over themself
Added hook 'trnnam_end_trans_hook' that is called when transaction ends
Added trn->used_tables that is used to an entry for all tables used by transaction
Fixed that ndb doesn't crash on duplicate key error when start_bulk_insert/end_bulk_insert are not called
include/maria.h:
Added argument to maria_end_bulk_insert() to know if the table will be deleted after the operation
include/my_tree.h:
Added macro 'reset_free_element()' to be able to ignore calls to the external free function.
Is used to optimize end-bulk-insert in case of failures, in which case we don't want write the remaining keys in the tree
mysql-test/install_test_db.sh:
Upgrade to new mysql_install_db options
mysql-test/r/maria-mvcc.result:
New tests
mysql-test/r/maria.result:
New tests
mysql-test/suite/ndb/r/ndb_auto_increment.result:
Fixed error message now when bulk insert is not always called
mysql-test/suite/ndb/t/ndb_auto_increment.test:
Fixed error message now when bulk insert is not always called
mysql-test/t/maria-mvcc.test:
Added testing of versioning of count(*)
mysql-test/t/maria-page-checksum.test:
Added comment
mysql-test/t/maria.test:
More tests
mysys/hash.c:
Code style change
sql/field.cc:
Allow storing year 2155 in year field
sql/ha_ndbcluster.cc:
Added new argument to end_bulk_insert() to signal if the bulk insert should ignored
sql/ha_ndbcluster.h:
Added new argument to end_bulk_insert() to signal if the bulk insert should ignored
sql/ha_partition.cc:
Added new argument to end_bulk_insert() to signal if the bulk insert should ignored
sql/ha_partition.h:
Added new argument to end_bulk_insert() to signal if the bulk insert should ignored
sql/handler.cc:
Don't call get_dup_key() if there is no table object. This can happen if the handler generates a duplicate key error on commit
sql/handler.h:
Added new argument to end_bulk_insert() to signal if the bulk insert should ignored (ie, the table will be deleted)
sql/item.cc:
Style fix
Removed compiler warning
sql/log_event.cc:
Added new argument to ha_end_bulk_insert()
sql/log_event_old.cc:
Added new argument to ha_end_bulk_insert()
sql/mysqld.cc:
Removed compiler warning
sql/protocol.cc:
Added DBUG
sql/sql_class.cc:
Added DBUG
Fixed wrong call to strmake
sql/sql_insert.cc:
Don't call bulk insert in case of inserting only one row (speed optimization as starting/stopping bulk insert involves a lot of if's)
Added new argument to ha_end_bulk_insert()
sql/sql_load.cc:
Added new argument to ha_end_bulk_insert()
sql/sql_parse.cc:
Style fixes
Avoid goto in common senario
sql/sql_select.cc:
When running with purify/valgrind avoid copying structures over themself. This is not a real bug in itself, but it's a waste of cycles and causes valgrind warnings
sql/sql_select.h:
Avoid copying structures over themself. This is not a real bug in itself, but it's a waste of cycles and causes valgrind warnings
sql/sql_table.cc:
Call HA_EXTRA_PREPARE_FOR_DROP if table created by ALTER TABLE is going to be dropped
Added new argument to ha_end_bulk_insert()
storage/archive/ha_archive.cc:
Added new argument to end_bulk_insert()
storage/archive/ha_archive.h:
Added new argument to end_bulk_insert()
storage/federated/ha_federated.cc:
Added new argument to end_bulk_insert()
storage/federated/ha_federated.h:
Added new argument to end_bulk_insert()
storage/maria/Makefile.am:
Added ma_state.c and ma_state.h
storage/maria/ha_maria.cc:
Versioning of count(*) and checksum
- share->state.state is now assumed to be correct, not handler->state
- Call _ma_setup_live_state() in external lock to get count(*)/checksum versioning. In case of
not versioned and not concurrent insertable table, file->s->state.state contains the correct state information
Other things:
- file->s -> share
- Added DBUG_ASSERT() for unlikely case
- Optimized end_bulk_insert() to not write anything if table is going to be deleted (as in failed alter table)
- Indentation changes in external_lock becasue of removed 'goto' caused a big conflict even if very little was changed
storage/maria/ha_maria.h:
New argument to end_bulk_insert()
storage/maria/ma_blockrec.c:
Update for versioning of count(*) and checksum
Keep share->state.state.data_file_length up to date (not info->state->data_file_length)
Moved _ma_block_xxxx_status() and maria_versioning() functions to ma_state.c
storage/maria/ma_check.c:
Update and use share->state.state instead of info->state
info->s to share
Update info->state at end of repair
Call _ma_reset_state() to update share->state_history at end of repair
storage/maria/ma_checkpoint.c:
Call _ma_remove_not_visible_states() on checkpoint to clean up not visible state history from tables
storage/maria/ma_close.c:
Remember state history for running transaction even if table is closed
storage/maria/ma_commit.c:
Ensure we always call trnman_commit_trn() even if other calls fails. If we don't do that, the translog and state structures will not be freed
storage/maria/ma_delete.c:
Versioning of count(*) and checksum:
- Always update info->state->checksum and info->state->records
storage/maria/ma_delete_all.c:
Versioning of count(*) and checksum:
- Ensure that share->state.state is updated, as here is where we store the primary information
storage/maria/ma_dynrec.c:
Use lock_key_trees instead of concurrent_insert to check if trees should be locked.
This allows us to lock trees both for concurrent_insert and for index versioning.
storage/maria/ma_extra.c:
Versioning of count(*) and checksum:
- Use share->state.state instead of info->state
- share->concurrent_insert -> share->non_transactional_concurrent_insert
- Don't update share->state.state from info->state if transactional table
Optimization:
- Don't flush io_cache or bitmap if we are using FLUSH_IGNORE_CHANGED
storage/maria/ma_info.c:
Get most state information from current state
storage/maria/ma_init.c:
Add hash table and free function to store states for closed tables
Install hook for transaction commit/rollback to update history state
storage/maria/ma_key_recover.c:
Versioning of count(*) and checksum:
- Use share->state.state instead of info->state
storage/maria/ma_locking.c:
Versioning of count(*) and checksum:
- Call virtual functions (if exists) to restore/update status
- Move _ma_xxx_status() functions to ma_state.c
info->s -> share
storage/maria/ma_open.c:
Versioning of count(*) and checksum:
- For not transactional tables, set info->state to point to new allocated state structure.
- Initialize new info->state_start variable that points to state at start of transaction
- Copy old history states from hash table (maria_stored_states) first time the table is opened
- Split flag share->concurrent_insert to non_transactional_concurrent_insert & lock_key_tree
- For now, only enable versioning of tables without keys (to be fixed in soon!)
- Added new virtual function to restore status in maria_lock_database)
More DBUG
storage/maria/ma_page.c:
Versioning of count(*) and checksum:
- Use share->state.state instead of info->state
- Modify share->state.state.key_file_length under share->intern_lock
storage/maria/ma_range.c:
Versioning of count(*) and checksum:
- Lock trees based on share->lock_key_trees
info->s -> share
storage/maria/ma_recovery.c:
Versioning of count(*) and checksum:
- Use share->state.state instead of info->state
- Update state information on close and when reenabling logging
storage/maria/ma_rkey.c:
Versioning of count(*) and checksum:
- Lock trees based on share->lock_key_trees
storage/maria/ma_rnext.c:
Versioning of count(*) and checksum:
- Lock trees based on share->lock_key_trees
storage/maria/ma_rnext_same.c:
Versioning of count(*) and checksum:
- Lock trees based on share->lock_key_trees
- Only skip rows based on file length if non_transactional_concurrent_insert is set
storage/maria/ma_rprev.c:
Versioning of count(*) and checksum:
- Lock trees based on share->lock_key_trees
storage/maria/ma_rsame.c:
Versioning of count(*) and checksum:
- Lock trees based on share->lock_key_trees
storage/maria/ma_sort.c:
Use share->state.state instead of info->state
Fixed indentation
storage/maria/ma_static.c:
Added maria_stored_state
storage/maria/ma_update.c:
Versioning of count(*) and checksum:
- Always update info->state->checksum and info->state->records
- Remove optimization for index file update as it doesn't work for transactional tables
storage/maria/ma_write.c:
Versioning of count(*) and checksum:
- Always update info->state->checksum and info->state->records
storage/maria/maria_def.h:
Move MARIA_STATUS_INFO to ma_state.h
Changes to MARIA_SHARE:
- Added state_history to store count(*)/checksum states
- Added in_trans as counter if table is used by running transactions
- Split concurrent_insert into lock_key_trees and on_transactional_concurrent_insert.
- Added virtual function lock_restore_status
Changes to MARIA_HA:
- save_state -> state_save
- Added state_start to store state at start of transaction
storage/maria/maria_pack.c:
Versioning of count(*) and checksum:
- Use share->state.state instead of info->state
Indentation fixes
storage/maria/trnman.c:
Added hook 'trnnam_end_trans_hook' that is called when transaction ends
Added trn->used_tables that is used to an entry for all tables used by transaction
More DBUG
Changed return type of trnman_end_trn() to my_bool
Added trnman_get_min_trid() to get minimum trid in use.
Added trnman_exists_active_transactions() to check if there exist a running transaction started between two commit id
storage/maria/trnman.h:
Added 'used_tables'
Moved all pointers into same groups to get better memory alignment
storage/maria/trnman_public.h:
Added prototypes for new functions and variables
Chagned return type of trnman_end_trn() to my_bool
storage/myisam/ha_myisam.cc:
Added argument to end_bulk_insert() if operation should be aborted
storage/myisam/ha_myisam.h:
Added argument to end_bulk_insert() if operation should be aborted
storage/maria/ma_state.c:
Functions to handle state of count(*) and checksum
storage/maria/ma_state.h:
Structures and declarations to handle state of count(*) and checksum
ha_partition::update_create_info() just calls update_create_info
of a first partition, so only get the autoincrement maximum
of the first partition, so SHOW CREATE TABLE can show
small AUTO_INCREMENT parameters.
Fixed by implementing ha_partition::update_create_info() in a way
other handlers work.
HA_ARCHIVE:stats.auto_increment handling made consistent with other engines
mysql-test/r/archive.result:
Bug #32247 Test reports wrong value of "AUTO_INCREMENT" (on a partitioned InnoDB table).
test result fixed
mysql-test/r/partition.result:
Bug #32247 Test reports wrong value of "AUTO_INCREMENT" (on a partitioned InnoDB table).
test result
mysql-test/suite/rpl/r/rpl_innodb_bug28430.result:
Bug #32247 Test reports wrong value of "AUTO_INCREMENT" (on a partitioned InnoDB table).
test result fixed
mysql-test/t/partition.test:
Bug #32247 Test reports wrong value of "AUTO_INCREMENT" (on a partitioned InnoDB table).
test added as rpl_innodb_bug28430 is INNOBASE specific while
the bug is not
sql/ha_partition.cc:
Bug #32247 Test reports wrong value of "AUTO_INCREMENT" (on a partitioned InnoDB table).
ha_partition::update_create_info() implemented
storage/archive/ha_archive.cc:
Bug #32247 Test reports wrong value of "AUTO_INCREMENT" (on a partitioned InnoDB table).
ha_archive.stats.auto_increment now contains next (not last used) value
of auto_increment to be consistent with other engines
