Base code and idea from a patch from by plinux at Taobao.
The idea is that we mark all memory that are thread specific with MY_THREAD_SPECIFIC.
Memory counting is done per thread in the my_malloc_size_cb_func callback function from my_malloc().
There are plenty of new asserts to ensure that for a debug server the counting is correct.
Information_schema.processlist gets two new columns: MEMORY_USED and EXAMINED_ROWS.
- The later is there mainly to show how query is progressing.
The following changes in interfaces was needed to get this to work:
- init_alloc_root() amd init_sql_alloc() has extra option so that one can mark memory with MY_THREAD_SPECIFIC
- One now have to use alloc_root_set_min_malloc() to set min memory to be allocated by alloc_root()
- my_init_dynamic_array() has extra option so that one can mark memory with MY_THREAD_SPECIFIC
- my_net_init() has extra option so that one can mark memory with MY_THREAD_SPECIFIC
- Added flag for hash_init() so that one can mark hash table to be thread specific.
- Added flags to init_tree() so that one can mark tree to be thread specific.
- Removed with_delete option to init_tree(). Now one should instead use MY_TREE_WITH_DELETE_FLAG.
- Added flag to Warning_info::Warning_info() if the structure should be fully initialized.
- String elements can now be marked as thread specific.
- Internal HEAP tables are now marking it's memory as MY_THREAD_SPECIFIC.
- Changed type of myf from int to ulong, as this is always a set of bit flags.
Other things:
- Removed calls to net_end() and thd->cleanup() as these are now done in ~THD()
- We now also show EXAMINED_ROWS in SHOW PROCESSLIST
- Added new variable 'memory_used'
- Fixed bug where kill_threads_for_user() was using the wrong mem_root to allocate memory.
- Removed calls to the obsoleted function init_dynamic_array()
- Use set_current_thd() instead of my_pthread_setspecific_ptr(THR_THD,...)
client/completion_hash.cc:
Updated call to init_alloc_root()
client/mysql.cc:
Updated call to init_alloc_root()
client/mysqlbinlog.cc:
init_dynamic_array() -> my_init_dynamic_array()
Updated call to init_alloc_root()
client/mysqlcheck.c:
Updated call to my_init_dynamic_array()
client/mysqldump.c:
Updated call to init_alloc_root()
client/mysqltest.cc:
Updated call to init_alloc_root()
Updated call to my_init_dynamic_array()
Fixed compiler warnings
extra/comp_err.c:
Updated call to my_init_dynamic_array()
extra/resolve_stack_dump.c:
Updated call to my_init_dynamic_array()
include/hash.h:
Added HASH_THREAD_SPECIFIC
include/heap.h:
Added flag is internal temporary table.
include/my_dir.h:
Safety fix: Ensure that MY_DONT_SORT and MY_WANT_STAT don't interfer with other mysys flags
include/my_global.h:
Changed type of myf from int to ulong, as this is always a set of bit flags.
include/my_sys.h:
Added MY_THREAD_SPECIFIC and MY_THREAD_MOVE
Added malloc_flags to DYNAMIC_ARRAY
Added extra mysys flag argument to my_init_dynamic_array()
Removed deprecated functions init_dynamic_array() and my_init_dynamic_array.._ci
Updated paramaters for init_alloc_root()
include/my_tree.h:
Added my_flags to allow one to use MY_THREAD_SPECIFIC with hash tables.
Removed with_delete. One should now instead use MY_TREE_WITH_DELETE_FLAG
Updated parameters to init_tree()
include/myisamchk.h:
Added malloc_flags to allow one to use MY_THREAD_SPECIFIC for checks.
include/mysql.h:
Added MYSQL_THREAD_SPECIFIC_MALLOC
Used 'unused1' to mark memory as thread specific.
include/mysql.h.pp:
Updated file
include/mysql_com.h:
Used 'unused1' to mark memory as thread specific.
Updated parameters for my_net_init()
libmysql/libmysql.c:
Updated call to init_alloc_root() to mark memory thread specific.
libmysqld/emb_qcache.cc:
Updated call to init_alloc_root()
libmysqld/lib_sql.cc:
Updated call to init_alloc_root()
mysql-test/r/create.result:
Updated results
mysql-test/r/user_var.result:
Updated results
mysql-test/suite/funcs_1/datadict/processlist_priv.inc:
Update to handle new format of SHOW PROCESSLIST
mysql-test/suite/funcs_1/datadict/processlist_val.inc:
Update to handle new format of SHOW PROCESSLIST
mysql-test/suite/funcs_1/r/is_columns_is.result:
Update to handle new format of SHOW PROCESSLIST
mysql-test/suite/funcs_1/r/processlist_priv_no_prot.result:
Updated results
mysql-test/suite/funcs_1/r/processlist_val_no_prot.result:
Updated results
mysql-test/t/show_explain.test:
Fixed usage of debug variable so that one can run test with --debug
mysql-test/t/user_var.test:
Added test of memory_usage variable.
mysys/array.c:
Added extra my_flags option to init_dynamic_array() and init_dynamic_array2() so that one can mark memory with MY_THREAD_SPECIFIC
All allocated memory is marked with the given my_flags.
Removed obsolete function init_dynamic_array()
mysys/default.c:
Updated call to init_alloc_root()
Updated call to my_init_dynamic_array()
mysys/hash.c:
Updated call to my_init_dynamic_array_ci().
Allocated memory is marked with MY_THREAD_SPECIFIC if HASH_THREAD_SPECIFIC is used.
mysys/ma_dyncol.c:
init_dynamic_array() -> my_init_dynamic_array()
Added #if to get rid of compiler warnings
mysys/mf_tempdir.c:
Updated call to my_init_dynamic_array()
mysys/my_alloc.c:
Added extra parameter to init_alloc_root() so that one can mark memory with MY_THREAD_SPECIFIC
Extend MEM_ROOT with a flag if memory is thread specific.
This is stored in block_size, to keep the size of the MEM_ROOT object identical as before.
Allocated memory is marked with MY_THREAD_SPECIFIC if used with init_alloc_root()
mysys/my_chmod.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_chsize.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_copy.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_create.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_delete.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_error.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_fopen.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_fstream.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_getwd.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_lib.c:
Updated call to init_alloc_root()
Updated call to my_init_dynamic_array()
Updated DBUG_PRINT because of change of myf type
mysys/my_lock.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_malloc.c:
Store at start of each allocated memory block the size of the block and if the block is thread specific.
Call malloc_size_cb_func, if set, with the memory allocated/freed.
Updated DBUG_PRINT because of change of myf type
mysys/my_open.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_pread.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_read.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_redel.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_rename.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_seek.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_sync.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_thr_init.c:
Ensure that one can call my_thread_dbug_id() even if thread is not properly initialized.
mysys/my_write.c:
Updated DBUG_PRINT because of change of myf type
mysys/mysys_priv.h:
Updated parameters to sf_malloc and sf_realloc()
mysys/safemalloc.c:
Added checking that for memory marked with MY_THREAD_SPECIFIC that it's the same thread that is allocation and freeing the memory.
Added sf_malloc_dbug_id() to allow MariaDB to specify which THD is handling the memory.
Added my_flags arguments to sf_malloc() and sf_realloc() to be able to mark memory with MY_THREAD_SPECIFIC.
Added sf_report_leaked_memory() to get list of memory not freed by a thread.
mysys/tree.c:
Added flags to init_tree() so that one can mark tree to be thread specific.
Removed with_delete option to init_tree(). Now one should instead use MY_TREE_WITH_DELETE_FLAG.
Updated call to init_alloc_root()
All allocated memory is marked with the given malloc flags
mysys/waiting_threads.c:
Updated call to my_init_dynamic_array()
sql-common/client.c:
Updated call to init_alloc_root() and my_net_init() to mark memory thread specific.
Updated call to my_init_dynamic_array().
Added MYSQL_THREAD_SPECIFIC_MALLOC so that client can mark memory as MY_THREAD_SPECIFIC.
sql-common/client_plugin.c:
Updated call to init_alloc_root()
sql/debug_sync.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
sql/event_scheduler.cc:
Removed calls to net_end() as this is now done in ~THD()
Call set_current_thd() to ensure that memory is assigned to right thread.
sql/events.cc:
my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/filesort.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
sql/filesort_utils.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
sql/ha_ndbcluster.cc:
Updated call to init_alloc_root()
Updated call to my_net_init()
Removed calls to net_end() and thd->cleanup() as these are now done in ~THD()
sql/ha_ndbcluster_binlog.cc:
Updated call to my_net_init()
Updated call to init_sql_alloc()
Removed calls to net_end() and thd->cleanup() as these are now done in ~THD()
sql/ha_partition.cc:
Updated call to init_alloc_root()
sql/handler.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
Added missing call to my_dir_end()
sql/item_func.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
sql/item_subselect.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
sql/item_sum.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
sql/log.cc:
More DBUG
Updated call to init_alloc_root()
sql/mdl.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
sql/mysqld.cc:
Added total_memory_used
Updated call to init_alloc_root()
Move mysql_cond_broadcast() before my_thread_end()
Added mariadb_dbug_id() to count memory per THD instead of per thread.
Added my_malloc_size_cb_func() callback function for my_malloc() to count memory.
Move initialization of mysqld_server_started and mysqld_server_initialized earlier.
Updated call to my_init_dynamic_array().
Updated call to my_net_init().
Call my_pthread_setspecific_ptr(THR_THD,...) to ensure that memory is assigned to right thread.
Added status variable 'memory_used'.
Updated call to init_alloc_root()
my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/mysqld.h:
Added set_current_thd()
sql/net_serv.cc:
Added new parameter to my_net_init() so that one can mark memory with MY_THREAD_SPECIFIC.
Store in net->thread_specific_malloc if memory is thread specific.
Mark memory to be thread specific if requested.
sql/opt_range.cc:
Updated call to my_init_dynamic_array()
Updated call to init_sql_alloc()
Added MY_THREAD_SPECIFIC to allocated memory.
sql/opt_subselect.cc:
Updated call to init_sql_alloc() to mark memory thread specific.
sql/protocol.cc:
Fixed compiler warning
sql/records.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
sql/rpl_filter.cc:
Updated call to my_init_dynamic_array()
sql/rpl_handler.cc:
Updated call to my_init_dynamic_array2()
sql/rpl_handler.h:
Updated call to init_sql_alloc()
sql/rpl_mi.cc:
Updated call to my_init_dynamic_array()
sql/rpl_tblmap.cc:
Updated call to init_alloc_root()
sql/rpl_utility.cc:
Updated call to my_init_dynamic_array()
sql/slave.cc:
Initialize things properly before calling functions that allocate memory.
Removed calls to net_end() as this is now done in ~THD()
sql/sp_head.cc:
Updated call to init_sql_alloc()
Updated call to my_init_dynamic_array()
Added parameter to warning_info() that it should be fully initialized.
sql/sp_pcontext.cc:
Updated call to my_init_dynamic_array()
sql/sql_acl.cc:
Updated call to init_sql_alloc()
Updated call to my_init_dynamic_array()
my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/sql_admin.cc:
Added parameter to warning_info() that it should be fully initialized.
sql/sql_analyse.h:
Updated call to init_tree() to mark memory thread specific.
sql/sql_array.h:
Updated call to my_init_dynamic_array() to mark memory thread specific.
sql/sql_audit.cc:
Updated call to my_init_dynamic_array()
sql/sql_base.cc:
Updated call to init_sql_alloc()
my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/sql_cache.cc:
Updated comment
sql/sql_class.cc:
Added parameter to warning_info() that not initialize it until THD is fully created.
Updated call to init_sql_alloc()
Mark THD::user_vars has to be thread specific.
Updated call to my_init_dynamic_array()
Ensure that memory allocated by THD is assigned to the THD.
More DBUG
Always acll net_end() in ~THD()
Assert that all memory signed to this THD is really deleted at ~THD.
Fixed set_status_var_init() to not reset memory_used.
my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/sql_class.h:
Added MY_THREAD_SPECIFIC to allocated memory.
Added malloc_size to THD to record allocated memory per THD.
sql/sql_delete.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
sql/sql_error.cc:
Added 'initialize' parameter to Warning_info() to say if should allocate memory for it's structures.
This is used by THD::THD() to not allocate memory until THD is ready.
Added Warning_info::free_memory()
sql/sql_error.h:
Updated Warning_info() class.
sql/sql_handler.cc:
Updated call to init_alloc_root() to mark memory thread specific.
sql/sql_insert.cc:
More DBUG
sql/sql_join_cache.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
sql/sql_lex.cc:
Updated call to my_init_dynamic_array()
sql/sql_lex.h:
Updated call to my_init_dynamic_array()
sql/sql_load.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
sql/sql_parse.cc:
Removed calls to net_end() and thd->cleanup() as these are now done in ~THD()
Ensure that examined_row_count() is reset before query.
Fixed bug where kill_threads_for_user() was using the wrong mem_root to allocate memory.
my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
Don't restore thd->status_var.memory_used when restoring thd->status_var
sql/sql_plugin.cc:
Updated call to init_alloc_root()
Updated call to my_init_dynamic_array()
Don't allocate THD on the stack, as this causes problems with valgrind when doing thd memory counting.
my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/sql_prepare.cc:
Added parameter to warning_info() that it should be fully initialized.
Updated call to init_sql_alloc() to mark memory thread specific.
sql/sql_reload.cc:
my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/sql_select.cc:
Updated call to my_init_dynamic_array() and init_sql_alloc() to mark memory thread specific.
Added MY_THREAD_SPECIFIC to allocated memory.
More DBUG
sql/sql_servers.cc:
Updated call to init_sql_alloc() to mark memory some memory thread specific.
my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/sql_show.cc:
Updated call to my_init_dynamic_array()
Mark my_dir() memory thread specific.
Use my_pthread_setspecific_ptr(THR_THD,...) to mark that allocated memory should be allocated to calling thread.
More DBUG.
Added malloc_size and examined_row_count to SHOW PROCESSLIST.
Added MY_THREAD_SPECIFIC to allocated memory.
Updated call to init_sql_alloc()
Added parameter to warning_info() that it should be fully initialized.
sql/sql_statistics.cc:
Fixed compiler warning
sql/sql_string.cc:
String elements can now be marked as thread specific.
sql/sql_string.h:
String elements can now be marked as thread specific.
sql/sql_table.cc:
Updated call to init_sql_alloc() and my_malloc() to mark memory thread specific
my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
Fixed compiler warning
sql/sql_test.cc:
Updated call to my_init_dynamic_array() to mark memory thread specific.
sql/sql_trigger.cc:
Updated call to init_sql_alloc()
sql/sql_udf.cc:
Updated call to init_sql_alloc()
my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/sql_update.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
sql/table.cc:
Updated call to init_sql_alloc().
Mark memory used by temporary tables, that are not for slave threads, as MY_THREAD_SPECIFIC
Updated call to init_sql_alloc()
sql/thr_malloc.cc:
Added my_flags argument to init_sql_alloc() to be able to mark memory as MY_THREAD_SPECIFIC.
sql/thr_malloc.h:
Updated prototype for init_sql_alloc()
sql/tztime.cc:
Updated call to init_sql_alloc()
Updated call to init_alloc_root() to mark memory thread specific.
my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/uniques.cc:
Updated calls to init_tree(), my_init_dynamic_array() and my_malloc() to mark memory thread specific.
sql/unireg.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
storage/csv/ha_tina.cc:
Updated call to init_alloc_root()
storage/federated/ha_federated.cc:
Updated call to init_alloc_root()
Updated call to my_init_dynamic_array()
Ensure that memory allocated by fedarated is registered for the system, not for the thread.
storage/federatedx/federatedx_io_mysql.cc:
Updated call to my_init_dynamic_array()
storage/federatedx/ha_federatedx.cc:
Updated call to init_alloc_root()
Updated call to my_init_dynamic_array()
storage/heap/ha_heap.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
storage/heap/heapdef.h:
Added parameter to hp_get_new_block() to be able to do thread specific memory tagging.
storage/heap/hp_block.c:
Added parameter to hp_get_new_block() to be able to do thread specific memory tagging.
storage/heap/hp_create.c:
- Internal HEAP tables are now marking it's memory as MY_THREAD_SPECIFIC.
- Use MY_TREE_WITH_DELETE instead of removed option 'with_delete'.
storage/heap/hp_open.c:
Internal HEAP tables are now marking it's memory as MY_THREAD_SPECIFIC.
storage/heap/hp_write.c:
Added new parameter to hp_get_new_block()
storage/maria/ma_bitmap.c:
Updated call to my_init_dynamic_array()
storage/maria/ma_blockrec.c:
Updated call to my_init_dynamic_array()
storage/maria/ma_check.c:
Updated call to init_alloc_root()
storage/maria/ma_ft_boolean_search.c:
Updated calls to init_tree() and init_alloc_root()
storage/maria/ma_ft_nlq_search.c:
Updated call to init_tree()
storage/maria/ma_ft_parser.c:
Updated call to init_tree()
Updated call to init_alloc_root()
storage/maria/ma_loghandler.c:
Updated call to my_init_dynamic_array()
storage/maria/ma_open.c:
Updated call to my_init_dynamic_array()
storage/maria/ma_sort.c:
Updated call to my_init_dynamic_array()
storage/maria/ma_write.c:
Updated calls to my_init_dynamic_array() and init_tree()
storage/maria/maria_pack.c:
Updated call to init_tree()
storage/maria/unittest/sequence_storage.c:
Updated call to my_init_dynamic_array()
storage/myisam/ft_boolean_search.c:
Updated call to init_tree()
Updated call to init_alloc_root()
storage/myisam/ft_nlq_search.c:
Updated call to init_tree()
storage/myisam/ft_parser.c:
Updated call to init_tree()
Updated call to init_alloc_root()
storage/myisam/ft_stopwords.c:
Updated call to init_tree()
storage/myisam/mi_check.c:
Updated call to init_alloc_root()
storage/myisam/mi_write.c:
Updated call to my_init_dynamic_array()
Updated call to init_tree()
storage/myisam/myisamlog.c:
Updated call to init_tree()
storage/myisam/myisampack.c:
Updated call to init_tree()
storage/myisam/sort.c:
Updated call to my_init_dynamic_array()
storage/myisammrg/ha_myisammrg.cc:
Updated call to init_sql_alloc()
storage/perfschema/pfs_check.cc:
Rest current_thd
storage/perfschema/pfs_instr.cc:
Removed DBUG_ENTER/DBUG_VOID_RETURN as at this point my_thread_var is not allocated anymore, which can cause problems.
support-files/compiler_warnings.supp:
Disable compiler warning from offsetof macro.
Generalized support for auto-updated and/or auto-initialized timestamp
and datetime columns. This patch is a reimplementation of MySQL's
"WL#5874: CURRENT_TIMESTAMP as DEFAULT for DATETIME columns". In order to
ease future merges, this implementation reused few function and variable
names from MySQL's patch, however the implementation is quite different.
TODO:
The only unresolved problem in this patch is the semantics of LOAD DATA for
TIMESTAMP and DATETIME columns in the cases when there are missing or NULL
columns. I couldn't fully comprehend the logic behind MySQL's behavior and
its relationship with their own documentation, so I left the results to be
more consistent with all other LOAD cases.
The problematic test cases can be seen by running the test file function_defaults,
and observing the test case differences. Those were left on purpose for discussion.
Now partition engine adds underlying tables to the QC and ask underlying tables engine permittion to cache the query and return result of the query.
Incorrect QC cleanup in case of table registration failure fixe.
Unified interface for myisammrg & partitioned engnes for QC.
sql/sql_insert.cc:
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
******
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
sql/sql_table.cc:
small cleanup
******
small cleanup
SHOW ALL PROBLEMS FOR MERGE TABLE COMPLIANCE IN 5.1".
The problem was that CHECK/REPAIR TABLE for a MERGE table which
had several children missing or in wrong engine reported only
issue with the first such table in its result-set. While in 5.0
this statement returned the whole list of problematic tables.
Ability to report problems for all children was lost during
significant refactorings of MERGE code which were done as part
of work on 5.1 and 5.5 releases.
This patch restores status quo ante refactorings by changing
code in such a way that:
1) Failure to open child table due to its absence during CHECK/
REPAIR TABLE for a MERGE table is not reported immediately
when its absence is discovered in open_tables(). Instead
handling/error reporting in such a situation is postponed
until the moment when children are attached.
2) Code performing attaching of children no longer stops when
it encounters first problem with one of the children during
CHECK/REPAIR TABLE. Instead it continues iteration through
the child list until all problems caused by child absence/
wrong engine are reported.
Note that even after this change problem with mismatch of
child/parent definition won't be reported if there is also
another child missing, but this is how it was in 5.0 as well.
mysql-test/r/merge.result:
Added test case for bug #11754210 - "45777: CHECK TABLE DOESN'T
SHOW ALL PROBLEMS FOR MERGE TABLE COMPLIANCE IN 5.1".
Adjusted results of existing tests to the fact that CHECK/REPAIR
TABLE statements now try to report problems about missing table/
wrong engine for all underlying tables, and to the fact that
mismatch of parent/child definitions is always reported as an
error and not a warning.
mysql-test/t/merge.test:
Added test case for bug #11754210 - "45777: CHECK TABLE DOESN'T
SHOW ALL PROBLEMS FOR MERGE TABLE COMPLIANCE IN 5.1".
sql/sql_base.cc:
Changed code responsible for opening tables to ignore the fact
that underlying tables of a MERGE table are missing, if this
table is opened for CHECK/REPAIR TABLE.
The absence of underlying tables in this case is now detected and
appropriate error is reported at the point when child tables are
attached. At this point we can produce full list of problematic
child tables/errors to be returned as part of CHECK/REPAIR TABLE
result-set.
storage/myisammrg/ha_myisammrg.cc:
Changed myisammrg_attach_children_callback() to handle new
situation, when during CHECK/REPAIR TABLE we do not report
error about missing child immediately when this fact is
discovered during open_tables() but postpone error-reporting
till the time when children are attached.
Also this callback is now responsible for pushing an error
mentioning problematic child table to the list of errors to
be reported by CHECK/REPAIR TABLE statements.
Finally, since now myrg_attach_children() no longer relies on
return value from callback to determine the end of the children
list, callback no longer needs to set my_errno value and can
be simplified.
Changed myrg_print_wrong_table() to always report a problem
with child table as an error and not as a warning. This makes
reporting for different types of issues with child tables
more consistent and compatible with 5.0 behavior.
storage/myisammrg/myrg_open.c:
Changed code in myrg_attach_children() not to abort on the
first problem with a child table when attaching children to
parent MERGE table during CHECK/REPAIR TABLE statement
execution. This allows CHECK/REPAIR TABLE to report problems
about absence/wrong engine for all underlying tables as
part of their result-set.
When executing row-ordered-retrieval index merge,
the handler was cloned, but it used the wrong
memory root, so instead of allocating memory
on the thread/query's mem_root, it used the table's
mem_root, resulting in non released memory in the
table object, and was not freed until the table was
closed.
Solution was to ensure that memory used during cloning
of a handler was allocated from the correct memory root.
This was implemented by fixing handler::clone() to also
take a name argument, so it can be used with partitioning.
And in ha_partition only allocate the ha_partition's ref, and
call the original ha_partition partitions clone() and set at cloned
partitions.
Fix of .bzrignore on Windows with VS 2010
Added locking of lock mutex when updating status in external_unlock() for Aria and MyISAM tables.
Fixed that 'source' command doesn't cause mysql command line tool to exit on error.
DEBUG_EXECUTE() and DEBUG_EVALUATE_IF() should not execute things based on wildcards. (Allows one to run --debug with mysql-test-run scripts that uses @debug)
Fixed several core dump, deadlock and crashed table bugs in handling of LOCK TABLE with MERGE tables:
- Added priority of locks to avoid crashes with MERGE tables.
- Added thr_lock_merge() to allow one to merge two results of thr_lock().
Fixed 'not found row' bug in REPLACE with Aria tables.
Mark MyISAM tables that are part of MERGE with HA_OPEN_MERGE_TABLE and set the locks to have priority THR_LOCK_MERGE_PRIV.
- By sorting MERGE tables last in thr_multi_unlock() it's safer to release and relock them many times (can happen when TRIGGERS are created)
Avoid printing (null) in debug file (to easier find out wrong NULL pointer usage with %s).
client/mysql.cc:
Fixed that 'source' command doesn't cause mysql command line tool to exit on error.
client/mysqltest.cc:
Don't send NULL to fn_format(). (Can cause crash on Solaris when using --debug)
dbug/dbug.c:
DEBUG_EXECUTE() and DEBUG_EVALUATE_IF() should not execute things based on wildcards.
include/my_base.h:
Added flag to signal if one opens a MERGE table.
Added extra() command to signal that one is not part of a MERGE table anymore.
include/thr_lock.h:
Added priority for locks (needed to fix bug in thr_lock when using MERGE tables)
Added option to thr_unlock() if get_status() should be called.
Added prototype for thr_merge_locks().
mysql-test/mysql-test-run.pl:
Ignore crashed table warnings for tables named 'crashed'.
mysql-test/r/merge.result:
Renamed triggers to make debugging easier.
Added some CHECK TABLES to catch errors earlier.
Additional tests.
mysql-test/r/merge_debug.result:
Test of error handling when reopening MERGE tables.
mysql-test/r/udf_query_cache.result:
Added missing flush status
mysql-test/suite/parts/r/partition_repair_myisam.result:
Update results
mysql-test/t/merge.test:
Renamed triggers to make debugging easier.
Added some CHECK TABLES to catch errors earlier.
Additional tests.
mysql-test/t/merge_debug.test:
Test of error handling when reopening MERGE tables.
mysql-test/t/udf_query_cache.test:
Added missing flush status
mysys/my_getopt.c:
Removed not used variable
mysys/my_symlink2.c:
Changed (null) to (NULL) to make it easier to find NULL arguments to DBUG_PRINT() functions.
(On linux, NULL to sprintf is printed 'null')
mysys/thr_lock.c:
Added priority of locks to avoid crashes with MERGE tables.
Added thr_lock_merge() to allow one to merge two results of thr_lock().
- This is needed for MyISAM as all locked table must share the same status. If not, you will not see newly inserted rows in other instances of the table.
If calling thr_unlock() with THR_UNLOCK_UPDATE_STATUS, call update_status() and restore_status() for the locks. This is needed in some rare cases where we call thr_unlock() followed by thr_lock() without calling external_unlock/external_lock in between.
Simplify loop in thr_multi_lock().
Added 'start_trans', which is called at end of thr_multi_lock() when all locks are taken.
- This was needed by Aria to ensure that transaction is started when we got all locks, not at get_status(). Without this, some rows could not be visible when we lock two tables at the same time, causing REPLACE using two tables to fail unexpectedly.
sql/handler.cc:
Add an assert() in handler::print_error() for "impossible errors" (like table is crashed) when --debug-assert-if-crashed-table is used.
sql/lock.cc:
Simplify mysql_lock_tables() code if get_lock_data() returns 0 locks.
Added new parameter to thr_multi_unlock()
In mysql_unlock_read_tables(), call first externa_unlock(), then thr_multi_unlock(); This is same order as we do in mysql_unlock_tables().
Don't abort locks in mysql_lock_abort() for merged tables when a MERGE table is deleted; Would cause a spin lock.
Added call to thr_merge_locks() in mysql_lock_merge() to ensure consistency in thr_locks().
- New locks of same type and table is stored after the old lock to ensure that we get the status from the original lock.
sql/mysql_priv.h:
Added debug_assert_if_crashed_table
sql/mysqld.cc:
Added --debug-assert-if-crashed-table
sql/parse_file.cc:
Don't print '(null)' in DBUG_PRINT of no dir given
sql/set_var.cc:
Increase default size of buffer for @debug variable.
sql/sql_base.cc:
In case of error from reopen_table() in reopen_tables(), call unlock_open_table() and restart loop.
- This fixed bug when we twice deleted same table from open_cache.
Don't take name lock for already name locked table in open_unireg_entry().
- Fixed bug when doing repair in reopen_table().
- In detach_merge_children(), always detach if 'clear_refs' is given. We can't trust parent->children_attached as this function can be called twice, first time with clear_refs set to 0.
sql/sql_class.cc:
Changed printing of (null) to "" in set_thd_proc_info()
sql/sql_parse.cc:
Added DBUG
sql/sql_trigger.cc:
Don't call unlink_open_table() if reopen_table() fails as the table may already be freed.
storage/maria/ma_bitmap.c:
Fixed DBUG_ASSERT() in allocate_tail()
storage/maria/ma_blockrec.c:
Fixed wrong calculation of row length for very small rows in undo_row_update().
- Fixes ASSERT() when doing undo.
storage/maria/ma_blockrec.h:
Added _ma_block_start_trans() and _ma_block_start_trans_no_versioning()
storage/maria/ma_locking.c:
Call _ma_update_status_with_lock() when releasing write locks.
- Fixes potential problem with updating status without the proper lock.
storage/maria/ma_open.c:
Changed to use start_trans() instead of get_status() to ensure that we see all rows in all locked tables when we got the locks.
- Fixed 'not found row' bug in REPLACE with Aria tables.
storage/maria/ma_state.c:
Added _ma_update_status_with_lock() and _ma_block_start_trans().
This is to ensure that we see all rows in all locked tables when we got the locks.
storage/maria/ma_state.h:
Added _ma_update_status_with_lock()
storage/maria/ma_write.c:
More DBUG_PRINT
storage/myisam/mi_check.c:
Fixed error message
storage/myisam/mi_extra.c:
Added HA_EXTRA_DETACH_CHILD:
- Detach MyISAM table to not be part of MERGE table (remove flag & lock priority).
storage/myisam/mi_locking.c:
Call mi_update_status_with_lock() when releasing write locks.
- Fixes potential problem with updating status without the proper lock.
Change to use new HA_OPEN_MERGE_TABLE flag to test if MERGE table.
Added mi_fix_status(), called by thr_merge().
storage/myisam/mi_open.c:
Added marker if part of MERGE table.
Call mi_fix_status() in thr_lock() for transactional tables.
storage/myisam/myisamdef.h:
Change my_once_flag to uint, as it stored different values than just 0/1
Added 'open_flag' to store state given to mi_open()
storage/myisammrg/ha_myisammrg.cc:
Add THR_LOCK_MERGE_PRIV to THR_LOCK_DATA to get MERGE locks sorted after other types of locks.
storage/myisammrg/myrg_locking.c:
Remove windows specific code.
storage/myisammrg/myrg_open.c:
Use HA_OPEN_MERGE_TABLE to mi_open().
Set HA_OPEN_MERGE_TABLE for linked MyISAM tables.
storage/xtradb/buf/buf0buf.c:
Fixed compiler warning
storage/xtradb/buf/buf0lru.c:
Initialize variable that could be used not initialized.
Fix assorted warnings that are generated in optimized builds.
Most of it is silencing variables that are set but unused.
This patch also introduces the MY_ASSERT_UNREACHABLE macro
which helps the compiler to deduce that a certain piece of
code is unreachable.
include/my_compiler.h:
Use GCC's __builtin_unreachable if available. It allows
GCC to deduce the unreachability of certain code paths,
thus avoiding warnings that, for example, accused that a
variable could be used without being initialized (due to
unreachable code paths).
Bug#54678: InnoDB, TRUNCATE, ALTER, I_S SELECT, crash or deadlock
- Incompatible change: truncate no longer resorts to a row by
row delete if the storage engine does not support the truncate
method. Consequently, the count of affected rows does not, in
any case, reflect the actual number of rows.
- Incompatible change: it is no longer possible to truncate a
table that participates as a parent in a foreign key constraint,
unless it is a self-referencing constraint (both parent and child
are in the same table). To work around this incompatible change
and still be able to truncate such tables, disable foreign checks
with SET foreign_key_checks=0 before truncate. Alternatively, if
foreign key checks are necessary, please use a DELETE statement
without a WHERE condition.
Problem description:
The problem was that for storage engines that do not support
truncate table via a external drop and recreate, such as InnoDB
which implements truncate via a internal drop and recreate, the
delete_all_rows method could be invoked with a shared metadata
lock, causing problems if the engine needed exclusive access
to some internal metadata. This problem originated with the
fact that there is no truncate specific handler method, which
ended up leading to a abuse of the delete_all_rows method that
is primarily used for delete operations without a condition.
Solution:
The solution is to introduce a truncate handler method that is
invoked when the engine does not support truncation via a table
drop and recreate. This method is invoked under a exclusive
metadata lock, so that there is only a single instance of the
table when the method is invoked.
Also, the method is not invoked and a error is thrown if
the table is a parent in a non-self-referencing foreign key
relationship. This was necessary to avoid inconsistency as
some integrity checks are bypassed. This is inline with the
fact that truncate is primarily a DDL operation that was
designed to quickly remove all data from a table.
mysql-test/suite/innodb/t/innodb-truncate.test:
Add test cases for truncate and foreign key checks.
Also test that InnoDB resets auto-increment on truncate.
mysql-test/suite/innodb/t/innodb.test:
FK is not necessary, test is related to auto-increment.
Update error number, truncate is no longer invoked if
table is parent in a FK relationship.
mysql-test/suite/innodb/t/innodb_mysql.test:
Update error number, truncate is no longer invoked if
table is parent in a FK relationship.
Use delete instead of truncate, test is used to check
the interaction of FKs, triggers and delete.
mysql-test/suite/parts/inc/partition_check.inc:
Fix typo.
mysql-test/suite/sys_vars/t/foreign_key_checks_func.test:
Update error number, truncate is no longer invoked if
table is parent in a FK relationship.
mysql-test/t/mdl_sync.test:
Modify test case to reflect and ensure that truncate takes
a exclusive metadata lock.
mysql-test/t/trigger-trans.test:
Update error number, truncate is no longer invoked if
table is parent in a FK relationship.
sql/ha_partition.cc:
Reorganize the various truncate methods. delete_all_rows is now
passed directly to the underlying engines, so as truncate. The
code responsible for truncating individual partitions is moved
to ha_partition::truncate_partition, which is invoked when a
ALTER TABLE t1 TRUNCATE PARTITION p statement is executed.
Since the partition truncate no longer can be invoked via
delete, the bitmap operations are not necessary anymore. The
explicit reset of the auto-increment value is also removed
as the underlying engines are now responsible for reseting
the value.
sql/handler.cc:
Wire up the handler truncate method.
sql/handler.h:
Introduce and document the truncate handler method. It assumes
certain use cases of delete_all_rows.
Add method to retrieve the list of foreign keys referencing a
table. Method is used to avoid truncating tables that are
parent in a foreign key relationship.
sql/share/errmsg-utf8.txt:
Add error message for truncate and FK.
sql/sql_lex.h:
Introduce a flag so that the partition engine can detect when
a partition is being truncated. Used to give a special error.
sql/sql_parse.cc:
Function mysql_truncate_table no longer exists.
sql/sql_partition_admin.cc:
Implement the TRUNCATE PARTITION statement.
sql/sql_truncate.cc:
Change the truncate table implementation to use the new truncate
handler method and to not rely on row-by-row delete anymore.
The truncate handler method is always invoked with a exclusive
metadata lock. Also, it is no longer possible to truncate a
table that is parent in some non-self-referencing foreign key.
storage/archive/ha_archive.cc:
Rename method as the description indicates that in the future
this could be a truncate operation.
storage/blackhole/ha_blackhole.cc:
Implement truncate as no operation for the blackhole engine in
order to remain compatible with older releases.
storage/federated/ha_federated.cc:
Introduce truncate method that invokes delete_all_rows.
This is required to support partition truncate as this
form of truncate does not implement the drop and recreate
protocol.
storage/heap/ha_heap.cc:
Introduce truncate method that invokes delete_all_rows.
This is required to support partition truncate as this
form of truncate does not implement the drop and recreate
protocol.
storage/ibmdb2i/ha_ibmdb2i.cc:
Introduce truncate method that invokes delete_all_rows.
This is required to support partition truncate as this
form of truncate does not implement the drop and recreate
protocol.
storage/innobase/handler/ha_innodb.cc:
Rename delete_all_rows to truncate. InnoDB now does truncate
under a exclusive metadata lock.
Introduce and reorganize methods used to retrieve the list
of foreign keys referenced by a or referencing a table.
storage/myisammrg/ha_myisammrg.cc:
Introduce truncate method that invokes delete_all_rows.
This is required in order to remain compatible with earlier
releases where truncate would resort to a row-by-row delete.
for ALTER TABLE + MERGE tables
The patch for Bug#56292 changed how metadata locks are taken for MERGE
tables. After the patch, locking the MERGE table will also lock the
children tables with the same metadata lock type. This means that
LOCK TABLES on a MERGE table also will implicitly do LOCK TABLES on
the children tables.
A consequence of this change, is that it is possible to do LOCK TABLES
on a child table both explicitly and implicitly with the same statement
and that these two locks can be of different strength. For example,
LOCK TABLES child READ, merge WRITE.
In LOCK TABLES mode, we are not allowed to take new locks and each
statement must therefore try to find an existing TABLE instance with
a suitable lock. The code that searched for a suitable TABLE instance,
only considered table level locks. If a child table was locked twice,
it was therefore possible for this code to find a TABLE instance with
suitable table level locks but without suitable metadata lock.
This problem caused the assert in upgrade_shared_lock_to_exclusive()
to be triggered as it tried to upgrade a MDL_SHARED lock to
EXCLUSIVE. The problem was a regression caused by the patch for
Bug#56292.
This patch fixes the problem by partially reverting the changes
done by Bug#56292. Now, the children tables will only use the
same metadata lock as the MERGE table for MDL_SHARED_NO_WRITE
when not in locked tables mode. This means that LOCK TABLE
on a MERGE table will not implicitly lock the children tables.
This still fixes the original problem in Bug#56292 without
causing a regression.
Test case added to merge.test.
ALTER TABLE on a MERGE table could cause a deadlock with two
other connections if we reached a situation where:
1) A connection doing ALTER TABLE can't upgrade to MDL_EXCLUSIVE on the
parent table, but holds TL_READ_NO_INSERT on the child tables.
2) A connection doing DELETE on a child table can't get TL_WRITE on it
since ALTER TABLE holds TL_READ_NO_INSERT.
3) A connection doing SELECT on the parent table can't get TL_READ on
the child tables since TL_WRITE is ahead in the lock queue, but holds
MDL_SHARED_READ on the parent table preventing ALTER TABLE from upgrading.
For regular tables, this deadlock is avoided by having ALTER TABLE
take a MDL_SHARED_NO_WRITE metadata lock on the table. This prevents
DELETE from acquiring MDL_SHARED_WRITE on the table before ALTER TABLE
tries to upgrade to MDL_EXCLUSIVE. In the example above, SELECT would
therefore not be blocked by the pending DELETE as DELETE would not be
able to enter TL_WRITE in the table lock queue.
This patch fixes the problem for merge tables by using the same metadata
lock type for child tables as for the parent table. The child tables will
in this case therefore be locked with MDL_SHARED_NO_WRITE, preventing
DELETE from acquiring a metadata lock and enter into the table lock queue.
Change in behavior: By taking the same metadata lock for child tables
as for the parent table, LOCK TABLE on the parent table will now also
implicitly lock the child tables. Since LOCK TABLE on the parent table
now takes more than one metadata lock, it is possible for LOCK TABLE
... WRITE on the parent table or child tables to give ER_LOCK_DEADLOCK
error.
Test case added to mdl_sync.test.
Merge.test/.result has been updated to reflect the change to LOCK TABLE.
- Changed to still use bcmp() in certain cases becasue
- Faster for short unaligneed strings than memcmp()
- Bettern when using valgrind
- Changed to use my_sprintf() instead of sprintf() to get higher portability for old systems
- Changed code to use MariaDB version of select->skip_record()
- Removed -%::SCCS/s.% from Makefile.am:s to remove automake warnings
