mariadb/storage/heap/ha_heap.cc
Michael Widenius a260b15554 MDEV-4011 Added per thread memory counting and usage
Base code and idea from a patch from by plinux at Taobao.

The idea is that we mark all memory that are thread specific with MY_THREAD_SPECIFIC.
Memory counting is done per thread in the my_malloc_size_cb_func callback function from my_malloc().
There are plenty of new asserts to ensure that for a debug server the counting is correct.

Information_schema.processlist gets two new columns: MEMORY_USED and EXAMINED_ROWS.
- The later is there mainly to show how query is progressing.

The following changes in interfaces was needed to get this to work:
- init_alloc_root() amd init_sql_alloc() has extra option so that one can mark memory with MY_THREAD_SPECIFIC
- One now have to use alloc_root_set_min_malloc() to set min memory to be allocated by alloc_root()
- my_init_dynamic_array()  has extra option so that one can mark memory with MY_THREAD_SPECIFIC
- my_net_init() has extra option so that one can mark memory with MY_THREAD_SPECIFIC
- Added flag for hash_init() so that one can mark hash table to be thread specific.
- Added flags to init_tree() so that one can mark tree to be thread specific.
- Removed with_delete option to init_tree(). Now one should instead use MY_TREE_WITH_DELETE_FLAG.
- Added flag to Warning_info::Warning_info() if the structure should be fully initialized.
- String elements can now be marked as thread specific.
- Internal HEAP tables are now marking it's memory as MY_THREAD_SPECIFIC.
- Changed type of myf from int to ulong, as this is always a set of bit flags.

Other things:
- Removed calls to net_end() and thd->cleanup() as these are now done in ~THD()
- We now also show EXAMINED_ROWS in SHOW PROCESSLIST
- Added new variable 'memory_used'
- Fixed bug where kill_threads_for_user() was using the wrong mem_root to allocate memory.
- Removed calls to the obsoleted function init_dynamic_array()
- Use set_current_thd() instead of my_pthread_setspecific_ptr(THR_THD,...)


client/completion_hash.cc:
  Updated call to init_alloc_root()
client/mysql.cc:
  Updated call to init_alloc_root()
client/mysqlbinlog.cc:
  init_dynamic_array() -> my_init_dynamic_array()
  Updated call to init_alloc_root()
client/mysqlcheck.c:
  Updated call to my_init_dynamic_array()
client/mysqldump.c:
  Updated call to init_alloc_root()
client/mysqltest.cc:
  Updated call to init_alloc_root()
  Updated call to my_init_dynamic_array()
  Fixed compiler warnings
extra/comp_err.c:
  Updated call to my_init_dynamic_array()
extra/resolve_stack_dump.c:
  Updated call to my_init_dynamic_array()
include/hash.h:
  Added HASH_THREAD_SPECIFIC
include/heap.h:
  Added flag is internal temporary table.
include/my_dir.h:
  Safety fix: Ensure that MY_DONT_SORT and MY_WANT_STAT don't interfer with other mysys flags
include/my_global.h:
  Changed type of myf from int to ulong, as this is always a set of bit flags.
include/my_sys.h:
  Added MY_THREAD_SPECIFIC and MY_THREAD_MOVE
  Added malloc_flags to DYNAMIC_ARRAY
  Added extra mysys flag argument to my_init_dynamic_array()
  Removed deprecated functions init_dynamic_array() and my_init_dynamic_array.._ci
  Updated paramaters for init_alloc_root()
include/my_tree.h:
  Added my_flags to allow one to use MY_THREAD_SPECIFIC with hash tables.
  Removed with_delete. One should now instead use MY_TREE_WITH_DELETE_FLAG
  Updated parameters to init_tree()
include/myisamchk.h:
  Added malloc_flags to allow one to use MY_THREAD_SPECIFIC for checks.
include/mysql.h:
  Added MYSQL_THREAD_SPECIFIC_MALLOC
  Used 'unused1' to mark memory as thread specific.
include/mysql.h.pp:
  Updated file
include/mysql_com.h:
  Used 'unused1' to mark memory as thread specific.
  Updated parameters for my_net_init()
libmysql/libmysql.c:
  Updated call to init_alloc_root() to mark memory thread specific.
libmysqld/emb_qcache.cc:
  Updated call to init_alloc_root()
libmysqld/lib_sql.cc:
  Updated call to init_alloc_root()
mysql-test/r/create.result:
  Updated results
mysql-test/r/user_var.result:
  Updated results
mysql-test/suite/funcs_1/datadict/processlist_priv.inc:
  Update to handle new format of SHOW PROCESSLIST
mysql-test/suite/funcs_1/datadict/processlist_val.inc:
  Update to handle new format of SHOW PROCESSLIST
mysql-test/suite/funcs_1/r/is_columns_is.result:
  Update to handle new format of SHOW PROCESSLIST
mysql-test/suite/funcs_1/r/processlist_priv_no_prot.result:
  Updated results
mysql-test/suite/funcs_1/r/processlist_val_no_prot.result:
  Updated results
mysql-test/t/show_explain.test:
  Fixed usage of debug variable so that one can run test with --debug
mysql-test/t/user_var.test:
  Added test of memory_usage variable.
mysys/array.c:
  Added extra my_flags option to init_dynamic_array() and init_dynamic_array2() so that one can mark memory with MY_THREAD_SPECIFIC
  All allocated memory is marked with the given my_flags.
  Removed obsolete function init_dynamic_array()
mysys/default.c:
  Updated call to init_alloc_root()
  Updated call to my_init_dynamic_array()
mysys/hash.c:
  Updated call to my_init_dynamic_array_ci().
  Allocated memory is marked with MY_THREAD_SPECIFIC if HASH_THREAD_SPECIFIC is used.
mysys/ma_dyncol.c:
  init_dynamic_array() -> my_init_dynamic_array()
  Added #if to get rid of compiler warnings
mysys/mf_tempdir.c:
  Updated call to my_init_dynamic_array()
mysys/my_alloc.c:
  Added extra parameter to init_alloc_root() so that one can mark memory with MY_THREAD_SPECIFIC
  Extend MEM_ROOT with a flag if memory is thread specific.
  This is stored in block_size, to keep the size of the MEM_ROOT object identical as before.
  Allocated memory is marked with MY_THREAD_SPECIFIC if used with init_alloc_root()
mysys/my_chmod.c:
  Updated DBUG_PRINT because of change of myf type
mysys/my_chsize.c:
  Updated DBUG_PRINT because of change of myf type
mysys/my_copy.c:
  Updated DBUG_PRINT because of change of myf type
mysys/my_create.c:
  Updated DBUG_PRINT because of change of myf type
mysys/my_delete.c:
  Updated DBUG_PRINT because of change of myf type
mysys/my_error.c:
  Updated DBUG_PRINT because of change of myf type
mysys/my_fopen.c:
  Updated DBUG_PRINT because of change of myf type
mysys/my_fstream.c:
  Updated DBUG_PRINT because of change of myf type
mysys/my_getwd.c:
  Updated DBUG_PRINT because of change of myf type
mysys/my_lib.c:
  Updated call to init_alloc_root()
  Updated call to my_init_dynamic_array()
  Updated DBUG_PRINT because of change of myf type
mysys/my_lock.c:
  Updated DBUG_PRINT because of change of myf type
mysys/my_malloc.c:
  Store at start of each allocated memory block the size of the block and if the block is thread specific.
  Call malloc_size_cb_func, if set, with the memory allocated/freed.
  Updated DBUG_PRINT because of change of myf type
mysys/my_open.c:
  Updated DBUG_PRINT because of change of myf type
mysys/my_pread.c:
  Updated DBUG_PRINT because of change of myf type
mysys/my_read.c:
  Updated DBUG_PRINT because of change of myf type
mysys/my_redel.c:
  Updated DBUG_PRINT because of change of myf type
mysys/my_rename.c:
  Updated DBUG_PRINT because of change of myf type
mysys/my_seek.c:
  Updated DBUG_PRINT because of change of myf type
mysys/my_sync.c:
  Updated DBUG_PRINT because of change of myf type
mysys/my_thr_init.c:
  Ensure that one can call my_thread_dbug_id() even if thread is not properly initialized.
mysys/my_write.c:
  Updated DBUG_PRINT because of change of myf type
mysys/mysys_priv.h:
  Updated parameters to sf_malloc and sf_realloc()
mysys/safemalloc.c:
  Added checking that for memory marked with MY_THREAD_SPECIFIC that it's the same thread that is allocation and freeing the memory.
  Added sf_malloc_dbug_id() to allow MariaDB to specify which THD is handling the memory.
  Added my_flags arguments to sf_malloc() and sf_realloc() to be able to mark memory with MY_THREAD_SPECIFIC.
  Added sf_report_leaked_memory() to get list of memory not freed by a thread.
mysys/tree.c:
  Added flags to init_tree() so that one can mark tree to be thread specific.
  Removed with_delete option to init_tree(). Now one should instead use MY_TREE_WITH_DELETE_FLAG.
  Updated call to init_alloc_root()
  All allocated memory is marked with the given malloc flags
mysys/waiting_threads.c:
  Updated call to my_init_dynamic_array()
sql-common/client.c:
  Updated call to init_alloc_root() and my_net_init() to mark memory thread specific.
  Updated call to my_init_dynamic_array().
  Added MYSQL_THREAD_SPECIFIC_MALLOC so that client can mark memory as MY_THREAD_SPECIFIC.
sql-common/client_plugin.c:
  Updated call to init_alloc_root()
sql/debug_sync.cc:
  Added MY_THREAD_SPECIFIC to allocated memory.
sql/event_scheduler.cc:
  Removed calls to net_end() as this is now done in ~THD()
  Call set_current_thd() to ensure that memory is assigned to right thread.
sql/events.cc:
  my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/filesort.cc:
  Added MY_THREAD_SPECIFIC to allocated memory.
sql/filesort_utils.cc:
  Added MY_THREAD_SPECIFIC to allocated memory.
sql/ha_ndbcluster.cc:
  Updated call to init_alloc_root()
  Updated call to my_net_init()
  Removed calls to net_end() and thd->cleanup() as these are now done in ~THD()
sql/ha_ndbcluster_binlog.cc:
  Updated call to my_net_init()
  Updated call to init_sql_alloc()
  Removed calls to net_end() and thd->cleanup() as these are now done in ~THD()
sql/ha_partition.cc:
  Updated call to init_alloc_root()
sql/handler.cc:
  Added MY_THREAD_SPECIFIC to allocated memory.
  Added missing call to my_dir_end()
sql/item_func.cc:
  Added MY_THREAD_SPECIFIC to allocated memory.
sql/item_subselect.cc:
  Added MY_THREAD_SPECIFIC to allocated memory.
sql/item_sum.cc:
  Added MY_THREAD_SPECIFIC to allocated memory.
sql/log.cc:
  More DBUG
  Updated call to init_alloc_root()
sql/mdl.cc:
  Added MY_THREAD_SPECIFIC to allocated memory.
sql/mysqld.cc:
  Added total_memory_used
  Updated call to init_alloc_root()
  Move mysql_cond_broadcast() before my_thread_end()
  Added mariadb_dbug_id() to count memory per THD instead of per thread.
  Added my_malloc_size_cb_func() callback function for my_malloc() to count memory.
  Move initialization of mysqld_server_started and mysqld_server_initialized earlier.
  Updated call to my_init_dynamic_array().
  Updated call to my_net_init().
  Call my_pthread_setspecific_ptr(THR_THD,...) to ensure that memory is assigned to right thread.
  Added status variable 'memory_used'.
  Updated call to init_alloc_root()
  my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/mysqld.h:
  Added set_current_thd()
sql/net_serv.cc:
  Added new parameter to my_net_init() so that one can mark memory with MY_THREAD_SPECIFIC.
  Store in net->thread_specific_malloc if memory is thread specific.
  Mark memory to be thread specific if requested.
sql/opt_range.cc:
  Updated call to my_init_dynamic_array()
  Updated call to init_sql_alloc()
  Added MY_THREAD_SPECIFIC to allocated memory.
sql/opt_subselect.cc:
  Updated call to init_sql_alloc() to mark memory thread specific.
sql/protocol.cc:
  Fixed compiler warning
sql/records.cc:
  Added MY_THREAD_SPECIFIC to allocated memory.
sql/rpl_filter.cc:
  Updated call to my_init_dynamic_array()
sql/rpl_handler.cc:
  Updated call to my_init_dynamic_array2()
sql/rpl_handler.h:
  Updated call to init_sql_alloc()
sql/rpl_mi.cc:
  Updated call to my_init_dynamic_array()
sql/rpl_tblmap.cc:
  Updated call to init_alloc_root()
sql/rpl_utility.cc:
  Updated call to my_init_dynamic_array()
sql/slave.cc:
  Initialize things properly before calling functions that allocate memory.
  Removed calls to net_end() as this is now done in ~THD()
sql/sp_head.cc:
  Updated call to init_sql_alloc()
  Updated call to my_init_dynamic_array()
  Added parameter to warning_info() that it should be fully initialized.
sql/sp_pcontext.cc:
  Updated call to my_init_dynamic_array()
sql/sql_acl.cc:
  Updated call to init_sql_alloc()
  Updated call to my_init_dynamic_array()
  my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/sql_admin.cc:
  Added parameter to warning_info() that it should be fully initialized.
sql/sql_analyse.h:
  Updated call to init_tree() to mark memory thread specific.
sql/sql_array.h:
  Updated call to my_init_dynamic_array() to mark memory thread specific.
sql/sql_audit.cc:
  Updated call to my_init_dynamic_array()
sql/sql_base.cc:
  Updated call to init_sql_alloc()
  my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/sql_cache.cc:
  Updated comment
sql/sql_class.cc:
  Added parameter to warning_info() that not initialize it until THD is fully created.
  Updated call to init_sql_alloc()
  Mark THD::user_vars has to be thread specific.
  Updated call to my_init_dynamic_array()
  Ensure that memory allocated by THD is assigned to the THD.
  More DBUG
  Always acll net_end() in ~THD()
  Assert that all memory signed to this THD is really deleted at ~THD.
  Fixed set_status_var_init() to not reset memory_used.
  my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/sql_class.h:
  Added MY_THREAD_SPECIFIC to allocated memory.
  Added malloc_size to THD to record allocated memory per THD.
sql/sql_delete.cc:
  Added MY_THREAD_SPECIFIC to allocated memory.
sql/sql_error.cc:
  Added 'initialize' parameter to Warning_info() to say if should allocate memory for it's structures.
  This is used by THD::THD() to not allocate memory until THD is ready.
  Added Warning_info::free_memory()
sql/sql_error.h:
  Updated Warning_info() class.
sql/sql_handler.cc:
  Updated call to init_alloc_root() to mark memory thread specific.
sql/sql_insert.cc:
  More DBUG
sql/sql_join_cache.cc:
  Added MY_THREAD_SPECIFIC to allocated memory.
sql/sql_lex.cc:
  Updated call to my_init_dynamic_array()
sql/sql_lex.h:
  Updated call to my_init_dynamic_array()
sql/sql_load.cc:
  Added MY_THREAD_SPECIFIC to allocated memory.
sql/sql_parse.cc:
  Removed calls to net_end() and thd->cleanup() as these are now done in ~THD()
  Ensure that examined_row_count() is reset before query.
  Fixed bug where kill_threads_for_user() was using the wrong mem_root to allocate memory.
  my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
  Don't restore thd->status_var.memory_used when restoring thd->status_var
sql/sql_plugin.cc:
  Updated call to init_alloc_root()
  Updated call to my_init_dynamic_array()
  Don't allocate THD on the stack, as this causes problems with valgrind when doing thd memory counting.
  my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/sql_prepare.cc:
  Added parameter to warning_info() that it should be fully initialized.
  Updated call to init_sql_alloc() to mark memory thread specific.
sql/sql_reload.cc:
  my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/sql_select.cc:
  Updated call to my_init_dynamic_array() and init_sql_alloc() to mark memory thread specific.
  Added MY_THREAD_SPECIFIC to allocated memory.
  More DBUG
sql/sql_servers.cc:
  Updated call to init_sql_alloc() to mark memory some memory thread specific.
  my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/sql_show.cc:
  Updated call to my_init_dynamic_array()
  Mark my_dir() memory thread specific.
  Use my_pthread_setspecific_ptr(THR_THD,...) to mark that allocated memory should be allocated to calling thread.
  More DBUG.
  Added malloc_size and examined_row_count to SHOW PROCESSLIST.
  Added MY_THREAD_SPECIFIC to allocated memory.
  Updated call to init_sql_alloc()
  Added parameter to warning_info() that it should be fully initialized.
sql/sql_statistics.cc:
  Fixed compiler warning
sql/sql_string.cc:
  String elements can now be marked as thread specific.
sql/sql_string.h:
  String elements can now be marked as thread specific.
sql/sql_table.cc:
  Updated call to init_sql_alloc() and my_malloc() to mark memory thread specific
  my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
  Fixed compiler warning
sql/sql_test.cc:
  Updated call to my_init_dynamic_array() to mark memory thread specific.
sql/sql_trigger.cc:
  Updated call to init_sql_alloc()
sql/sql_udf.cc:
  Updated call to init_sql_alloc()
  my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/sql_update.cc:
  Added MY_THREAD_SPECIFIC to allocated memory.
sql/table.cc:
  Updated call to init_sql_alloc().
  Mark memory used by temporary tables, that are not for slave threads, as MY_THREAD_SPECIFIC
  Updated call to init_sql_alloc()
sql/thr_malloc.cc:
  Added my_flags argument to init_sql_alloc() to be able to mark memory as MY_THREAD_SPECIFIC.
sql/thr_malloc.h:
  Updated prototype for init_sql_alloc()
sql/tztime.cc:
  Updated call to init_sql_alloc()
  Updated call to init_alloc_root() to mark memory thread specific.
  my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/uniques.cc:
  Updated calls to init_tree(), my_init_dynamic_array() and my_malloc() to mark memory thread specific.
sql/unireg.cc:
  Added MY_THREAD_SPECIFIC to allocated memory.
storage/csv/ha_tina.cc:
  Updated call to init_alloc_root()
storage/federated/ha_federated.cc:
  Updated call to init_alloc_root()
  Updated call to my_init_dynamic_array()
  Ensure that memory allocated by fedarated is registered for the system, not for the thread.
storage/federatedx/federatedx_io_mysql.cc:
  Updated call to my_init_dynamic_array()
storage/federatedx/ha_federatedx.cc:
  Updated call to init_alloc_root()
  Updated call to my_init_dynamic_array()
storage/heap/ha_heap.cc:
  Added MY_THREAD_SPECIFIC to allocated memory.
storage/heap/heapdef.h:
  Added parameter to hp_get_new_block() to be able to do thread specific memory tagging.
storage/heap/hp_block.c:
  Added parameter to hp_get_new_block() to be able to do thread specific memory tagging.
storage/heap/hp_create.c:
  - Internal HEAP tables are now marking it's memory as MY_THREAD_SPECIFIC.
  - Use MY_TREE_WITH_DELETE instead of removed option 'with_delete'.
storage/heap/hp_open.c:
  Internal HEAP tables are now marking it's memory as MY_THREAD_SPECIFIC.
storage/heap/hp_write.c:
  Added new parameter to hp_get_new_block()
storage/maria/ma_bitmap.c:
  Updated call to my_init_dynamic_array()
storage/maria/ma_blockrec.c:
  Updated call to my_init_dynamic_array()
storage/maria/ma_check.c:
  Updated call to init_alloc_root()
storage/maria/ma_ft_boolean_search.c:
  Updated calls to init_tree() and init_alloc_root()
storage/maria/ma_ft_nlq_search.c:
  Updated call to init_tree()
storage/maria/ma_ft_parser.c:
  Updated call to init_tree()
  Updated call to init_alloc_root()
storage/maria/ma_loghandler.c:
  Updated call to my_init_dynamic_array()
storage/maria/ma_open.c:
  Updated call to my_init_dynamic_array()
storage/maria/ma_sort.c:
  Updated call to my_init_dynamic_array()
storage/maria/ma_write.c:
  Updated calls to my_init_dynamic_array() and init_tree()
storage/maria/maria_pack.c:
  Updated call to init_tree()
storage/maria/unittest/sequence_storage.c:
  Updated call to my_init_dynamic_array()
storage/myisam/ft_boolean_search.c:
  Updated call to init_tree()
  Updated call to init_alloc_root()
storage/myisam/ft_nlq_search.c:
  Updated call to init_tree()
storage/myisam/ft_parser.c:
  Updated call to init_tree()
  Updated call to init_alloc_root()
storage/myisam/ft_stopwords.c:
  Updated call to init_tree()
storage/myisam/mi_check.c:
  Updated call to init_alloc_root()
storage/myisam/mi_write.c:
  Updated call to my_init_dynamic_array()
  Updated call to init_tree()
storage/myisam/myisamlog.c:
  Updated call to init_tree()
storage/myisam/myisampack.c:
  Updated call to init_tree()
storage/myisam/sort.c:
  Updated call to my_init_dynamic_array()
storage/myisammrg/ha_myisammrg.cc:
  Updated call to init_sql_alloc()
storage/perfschema/pfs_check.cc:
  Rest current_thd
storage/perfschema/pfs_instr.cc:
  Removed DBUG_ENTER/DBUG_VOID_RETURN as at this point my_thread_var is not allocated anymore, which can cause problems.
support-files/compiler_warnings.supp:
  Disable compiler warning from offsetof macro.
2013-01-23 16:16:14 +01:00

835 lines
22 KiB
C++

/*
Copyright (c) 2000, 2011, Oracle and/or its affiliates
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; version 2 of the License.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */
#ifdef USE_PRAGMA_IMPLEMENTATION
#pragma implementation // gcc: Class implementation
#endif
#define MYSQL_SERVER 1
#include "sql_priv.h"
#include "sql_plugin.h"
#include "ha_heap.h"
#include "heapdef.h"
#include "sql_base.h" // enum_tdc_remove_table_type
static handler *heap_create_handler(handlerton *hton,
TABLE_SHARE *table,
MEM_ROOT *mem_root);
static int
heap_prepare_hp_create_info(TABLE *table_arg, bool internal_table,
HP_CREATE_INFO *hp_create_info);
int heap_panic(handlerton *hton, ha_panic_function flag)
{
return hp_panic(flag);
}
int heap_init(void *p)
{
handlerton *heap_hton;
#ifdef HAVE_PSI_INTERFACE
init_heap_psi_keys();
#endif
heap_hton= (handlerton *)p;
heap_hton->state= SHOW_OPTION_YES;
heap_hton->db_type= DB_TYPE_HEAP;
heap_hton->create= heap_create_handler;
heap_hton->panic= heap_panic;
heap_hton->flags= HTON_CAN_RECREATE;
return 0;
}
static handler *heap_create_handler(handlerton *hton,
TABLE_SHARE *table,
MEM_ROOT *mem_root)
{
return new (mem_root) ha_heap(hton, table);
}
/*****************************************************************************
** HEAP tables
*****************************************************************************/
ha_heap::ha_heap(handlerton *hton, TABLE_SHARE *table_arg)
:handler(hton, table_arg), file(0), records_changed(0), key_stat_version(0),
internal_table(0)
{}
static const char *ha_heap_exts[] = {
NullS
};
const char **ha_heap::bas_ext() const
{
return ha_heap_exts;
}
/*
Hash index statistics is updated (copied from HP_KEYDEF::hash_buckets to
rec_per_key) after 1/HEAP_STATS_UPDATE_THRESHOLD fraction of table records
have been inserted/updated/deleted. delete_all_rows() and table flush cause
immediate update.
NOTE
hash index statistics must be updated when number of table records changes
from 0 to non-zero value and vice versa. Otherwise records_in_range may
erroneously return 0 and 'range' may miss records.
*/
#define HEAP_STATS_UPDATE_THRESHOLD 10
int ha_heap::open(const char *name, int mode, uint test_if_locked)
{
internal_table= test(test_if_locked & HA_OPEN_INTERNAL_TABLE);
if (internal_table || (!(file= heap_open(name, mode)) && my_errno == ENOENT))
{
HP_CREATE_INFO create_info;
my_bool created_new_share;
int rc;
file= 0;
if (heap_prepare_hp_create_info(table, internal_table, &create_info))
goto end;
create_info.pin_share= TRUE;
rc= heap_create(name, &create_info, &internal_share, &created_new_share);
my_free(create_info.keydef);
if (rc)
goto end;
implicit_emptied= test(created_new_share);
if (internal_table)
file= heap_open_from_share(internal_share, mode);
else
file= heap_open_from_share_and_register(internal_share, mode);
if (!file)
{
heap_release_share(internal_share, internal_table);
goto end;
}
}
ref_length= sizeof(HEAP_PTR);
/* Initialize variables for the opened table */
set_keys_for_scanning();
/*
We cannot run update_key_stats() here because we do not have a
lock on the table. The 'records' count might just be changed
temporarily at this moment and we might get wrong statistics (Bug
#10178). Instead we request for update. This will be done in
ha_heap::info(), which is always called before key statistics are
used.
*/
key_stat_version= file->s->key_stat_version-1;
end:
return (file ? 0 : 1);
}
int ha_heap::close(void)
{
return internal_table ? hp_close(file) : heap_close(file);
}
/*
Create a copy of this table
DESCRIPTION
Do same as default implementation but use file->s->name instead of
table->s->path. This is needed by Windows where the clone() call sees
'/'-delimited path in table->s->path, while ha_heap::open() was called
with '\'-delimited path.
*/
handler *ha_heap::clone(const char *name, MEM_ROOT *mem_root)
{
handler *new_handler= get_new_handler(table->s, mem_root, table->s->db_type());
if (new_handler && !new_handler->ha_open(table, file->s->name, table->db_stat,
HA_OPEN_IGNORE_IF_LOCKED))
return new_handler;
return NULL; /* purecov: inspected */
}
/*
Compute which keys to use for scanning
SYNOPSIS
set_keys_for_scanning()
no parameter
DESCRIPTION
Set the bitmap btree_keys, which is used when the upper layers ask
which keys to use for scanning. For each btree index the
corresponding bit is set.
RETURN
void
*/
void ha_heap::set_keys_for_scanning(void)
{
btree_keys.clear_all();
for (uint i= 0 ; i < table->s->keys ; i++)
{
if (table->key_info[i].algorithm == HA_KEY_ALG_BTREE)
btree_keys.set_bit(i);
}
}
int ha_heap::can_continue_handler_scan()
{
int error= 0;
if ((file->key_version != file->s->key_version && inited == INDEX) ||
(file->file_version != file->s->file_version && inited == RND))
{
/* Data changed, not safe to do index or rnd scan */
error= HA_ERR_RECORD_CHANGED;
}
return error;
}
void ha_heap::update_key_stats()
{
for (uint i= 0; i < table->s->keys; i++)
{
KEY *key=table->key_info+i;
if (!key->rec_per_key)
continue;
if (key->algorithm != HA_KEY_ALG_BTREE)
{
if (key->flags & HA_NOSAME)
key->rec_per_key[key->key_parts-1]= 1;
else
{
ha_rows hash_buckets= file->s->keydef[i].hash_buckets;
uint no_records= hash_buckets ? (uint) (file->s->records/hash_buckets) : 2;
if (no_records < 2)
no_records= 2;
key->rec_per_key[key->key_parts-1]= no_records;
}
}
}
records_changed= 0;
/* At the end of update_key_stats() we can proudly claim they are OK. */
key_stat_version= file->s->key_stat_version;
}
int ha_heap::write_row(uchar * buf)
{
int res;
if (table->next_number_field && buf == table->record[0])
{
if ((res= update_auto_increment()))
return res;
}
res= heap_write(file,buf);
if (!res && (++records_changed*HEAP_STATS_UPDATE_THRESHOLD >
file->s->records))
{
/*
We can perform this safely since only one writer at the time is
allowed on the table.
*/
file->s->key_stat_version++;
}
return res;
}
int ha_heap::update_row(const uchar * old_data, uchar * new_data)
{
int res;
res= heap_update(file,old_data,new_data);
if (!res && ++records_changed*HEAP_STATS_UPDATE_THRESHOLD >
file->s->records)
{
/*
We can perform this safely since only one writer at the time is
allowed on the table.
*/
file->s->key_stat_version++;
}
return res;
}
int ha_heap::delete_row(const uchar * buf)
{
int res;
res= heap_delete(file,buf);
if (!res && table->s->tmp_table == NO_TMP_TABLE &&
++records_changed*HEAP_STATS_UPDATE_THRESHOLD > file->s->records)
{
/*
We can perform this safely since only one writer at the time is
allowed on the table.
*/
file->s->key_stat_version++;
}
return res;
}
int ha_heap::index_read_map(uchar *buf, const uchar *key,
key_part_map keypart_map,
enum ha_rkey_function find_flag)
{
DBUG_ASSERT(inited==INDEX);
int error = heap_rkey(file,buf,active_index, key, keypart_map, find_flag);
table->status = error ? STATUS_NOT_FOUND : 0;
return error;
}
int ha_heap::index_read_last_map(uchar *buf, const uchar *key,
key_part_map keypart_map)
{
DBUG_ASSERT(inited==INDEX);
int error= heap_rkey(file, buf, active_index, key, keypart_map,
HA_READ_PREFIX_LAST);
table->status= error ? STATUS_NOT_FOUND : 0;
return error;
}
int ha_heap::index_read_idx_map(uchar *buf, uint index, const uchar *key,
key_part_map keypart_map,
enum ha_rkey_function find_flag)
{
int error = heap_rkey(file, buf, index, key, keypart_map, find_flag);
table->status = error ? STATUS_NOT_FOUND : 0;
return error;
}
int ha_heap::index_next(uchar * buf)
{
DBUG_ASSERT(inited==INDEX);
int error=heap_rnext(file,buf);
table->status=error ? STATUS_NOT_FOUND: 0;
return error;
}
int ha_heap::index_prev(uchar * buf)
{
DBUG_ASSERT(inited==INDEX);
int error=heap_rprev(file,buf);
table->status=error ? STATUS_NOT_FOUND: 0;
return error;
}
int ha_heap::index_first(uchar * buf)
{
DBUG_ASSERT(inited==INDEX);
int error=heap_rfirst(file, buf, active_index);
table->status=error ? STATUS_NOT_FOUND: 0;
return error;
}
int ha_heap::index_last(uchar * buf)
{
DBUG_ASSERT(inited==INDEX);
int error=heap_rlast(file, buf, active_index);
table->status=error ? STATUS_NOT_FOUND: 0;
return error;
}
int ha_heap::rnd_init(bool scan)
{
return scan ? heap_scan_init(file) : 0;
}
int ha_heap::rnd_next(uchar *buf)
{
int error=heap_scan(file, buf);
table->status=error ? STATUS_NOT_FOUND: 0;
return error;
}
int ha_heap::rnd_pos(uchar * buf, uchar *pos)
{
int error;
HEAP_PTR heap_position;
memcpy(&heap_position, pos, sizeof(HEAP_PTR));
error=heap_rrnd(file, buf, heap_position);
table->status=error ? STATUS_NOT_FOUND: 0;
return error;
}
void ha_heap::position(const uchar *record)
{
*(HEAP_PTR*) ref= heap_position(file); // Ref is aligned
}
int ha_heap::info(uint flag)
{
HEAPINFO hp_info;
if (!table)
return 1;
(void) heap_info(file,&hp_info,flag);
errkey= hp_info.errkey;
stats.records= hp_info.records;
stats.deleted= hp_info.deleted;
stats.mean_rec_length= hp_info.reclength;
stats.data_file_length= hp_info.data_length;
stats.index_file_length= hp_info.index_length;
stats.max_data_file_length= hp_info.max_records * hp_info.reclength;
stats.delete_length= hp_info.deleted * hp_info.reclength;
stats.create_time= (ulong) hp_info.create_time;
if (flag & HA_STATUS_AUTO)
stats.auto_increment_value= hp_info.auto_increment;
/*
If info() is called for the first time after open(), we will still
have to update the key statistics. Hoping that a table lock is now
in place.
*/
if (key_stat_version != file->s->key_stat_version)
update_key_stats();
return 0;
}
int ha_heap::extra(enum ha_extra_function operation)
{
return heap_extra(file,operation);
}
int ha_heap::reset()
{
return heap_reset(file);
}
int ha_heap::delete_all_rows()
{
heap_clear(file);
if (table->s->tmp_table == NO_TMP_TABLE)
{
/*
We can perform this safely since only one writer at the time is
allowed on the table.
*/
file->s->key_stat_version++;
}
return 0;
}
int ha_heap::reset_auto_increment(ulonglong value)
{
file->s->auto_increment= value;
return 0;
}
int ha_heap::external_lock(THD *thd, int lock_type)
{
return 0; // No external locking
}
/*
Disable indexes.
SYNOPSIS
disable_indexes()
mode mode of operation:
HA_KEY_SWITCH_NONUNIQ disable all non-unique keys
HA_KEY_SWITCH_ALL disable all keys
HA_KEY_SWITCH_NONUNIQ_SAVE dis. non-uni. and make persistent
HA_KEY_SWITCH_ALL_SAVE dis. all keys and make persistent
DESCRIPTION
Disable indexes and clear keys to use for scanning.
IMPLEMENTATION
HA_KEY_SWITCH_NONUNIQ is not implemented.
HA_KEY_SWITCH_NONUNIQ_SAVE is not implemented with HEAP.
HA_KEY_SWITCH_ALL_SAVE is not implemented with HEAP.
RETURN
0 ok
HA_ERR_WRONG_COMMAND mode not implemented.
*/
int ha_heap::disable_indexes(uint mode)
{
int error;
if (mode == HA_KEY_SWITCH_ALL)
{
if (!(error= heap_disable_indexes(file)))
set_keys_for_scanning();
}
else
{
/* mode not implemented */
error= HA_ERR_WRONG_COMMAND;
}
return error;
}
/*
Enable indexes.
SYNOPSIS
enable_indexes()
mode mode of operation:
HA_KEY_SWITCH_NONUNIQ enable all non-unique keys
HA_KEY_SWITCH_ALL enable all keys
HA_KEY_SWITCH_NONUNIQ_SAVE en. non-uni. and make persistent
HA_KEY_SWITCH_ALL_SAVE en. all keys and make persistent
DESCRIPTION
Enable indexes and set keys to use for scanning.
The indexes might have been disabled by disable_index() before.
The function works only if both data and indexes are empty,
since the heap storage engine cannot repair the indexes.
To be sure, call handler::delete_all_rows() before.
IMPLEMENTATION
HA_KEY_SWITCH_NONUNIQ is not implemented.
HA_KEY_SWITCH_NONUNIQ_SAVE is not implemented with HEAP.
HA_KEY_SWITCH_ALL_SAVE is not implemented with HEAP.
RETURN
0 ok
HA_ERR_CRASHED data or index is non-empty. Delete all rows and retry.
HA_ERR_WRONG_COMMAND mode not implemented.
*/
int ha_heap::enable_indexes(uint mode)
{
int error;
if (mode == HA_KEY_SWITCH_ALL)
{
if (!(error= heap_enable_indexes(file)))
set_keys_for_scanning();
}
else
{
/* mode not implemented */
error= HA_ERR_WRONG_COMMAND;
}
return error;
}
/*
Test if indexes are disabled.
SYNOPSIS
indexes_are_disabled()
no parameters
RETURN
0 indexes are not disabled
1 all indexes are disabled
[2 non-unique indexes are disabled - NOT YET IMPLEMENTED]
*/
int ha_heap::indexes_are_disabled(void)
{
return heap_indexes_are_disabled(file);
}
THR_LOCK_DATA **ha_heap::store_lock(THD *thd,
THR_LOCK_DATA **to,
enum thr_lock_type lock_type)
{
if (lock_type != TL_IGNORE && file->lock.type == TL_UNLOCK)
file->lock.type=lock_type;
*to++= &file->lock;
return to;
}
/*
We have to ignore ENOENT entries as the HEAP table is created on open and
not when doing a CREATE on the table.
*/
int ha_heap::delete_table(const char *name)
{
int error= heap_delete_table(name);
return error == ENOENT ? 0 : error;
}
void ha_heap::drop_table(const char *name)
{
file->s->delete_on_close= 1;
ha_close();
}
int ha_heap::rename_table(const char * from, const char * to)
{
return heap_rename(from,to);
}
ha_rows ha_heap::records_in_range(uint inx, key_range *min_key,
key_range *max_key)
{
KEY *key=table->key_info+inx;
if (key->algorithm == HA_KEY_ALG_BTREE)
return hp_rb_records_in_range(file, inx, min_key, max_key);
if (!min_key || !max_key ||
min_key->length != max_key->length ||
min_key->length != key->key_length ||
min_key->flag != HA_READ_KEY_EXACT ||
max_key->flag != HA_READ_AFTER_KEY)
return HA_POS_ERROR; // Can only use exact keys
if (stats.records <= 1)
return stats.records;
/* Assert that info() did run. We need current statistics here. */
DBUG_ASSERT(key_stat_version == file->s->key_stat_version);
return key->rec_per_key[key->key_parts-1];
}
static int
heap_prepare_hp_create_info(TABLE *table_arg, bool internal_table,
HP_CREATE_INFO *hp_create_info)
{
uint key, parts, mem_per_row= 0, keys= table_arg->s->keys;
uint auto_key= 0, auto_key_type= 0;
ha_rows max_rows;
HP_KEYDEF *keydef;
HA_KEYSEG *seg;
TABLE_SHARE *share= table_arg->s;
bool found_real_auto_increment= 0;
bzero(hp_create_info, sizeof(*hp_create_info));
for (key= parts= 0; key < keys; key++)
parts+= table_arg->key_info[key].key_parts;
if (!(keydef= (HP_KEYDEF*) my_malloc(keys * sizeof(HP_KEYDEF) +
parts * sizeof(HA_KEYSEG),
MYF(MY_WME | MY_THREAD_SPECIFIC))))
return my_errno;
seg= reinterpret_cast<HA_KEYSEG*>(keydef + keys);
for (key= 0; key < keys; key++)
{
KEY *pos= table_arg->key_info+key;
KEY_PART_INFO *key_part= pos->key_part;
KEY_PART_INFO *key_part_end= key_part + pos->key_parts;
keydef[key].keysegs= (uint) pos->key_parts;
keydef[key].flag= (pos->flags & (HA_NOSAME | HA_NULL_ARE_EQUAL));
keydef[key].seg= seg;
switch (pos->algorithm) {
case HA_KEY_ALG_UNDEF:
case HA_KEY_ALG_HASH:
keydef[key].algorithm= HA_KEY_ALG_HASH;
mem_per_row+= sizeof(char*) * 2; // = sizeof(HASH_INFO)
break;
case HA_KEY_ALG_BTREE:
keydef[key].algorithm= HA_KEY_ALG_BTREE;
mem_per_row+=sizeof(TREE_ELEMENT)+pos->key_length+sizeof(char*);
break;
default:
DBUG_ASSERT(0); // cannot happen
}
for (; key_part != key_part_end; key_part++, seg++)
{
Field *field= key_part->field;
if (pos->algorithm == HA_KEY_ALG_BTREE)
seg->type= field->key_type();
else
{
if ((seg->type = field->key_type()) != (int) HA_KEYTYPE_TEXT &&
seg->type != HA_KEYTYPE_VARTEXT1 &&
seg->type != HA_KEYTYPE_VARTEXT2 &&
seg->type != HA_KEYTYPE_VARBINARY1 &&
seg->type != HA_KEYTYPE_VARBINARY2 &&
seg->type != HA_KEYTYPE_BIT)
seg->type= HA_KEYTYPE_BINARY;
}
seg->start= (uint) key_part->offset;
seg->length= (uint) key_part->length;
seg->flag= key_part->key_part_flag;
if (field->flags & (ENUM_FLAG | SET_FLAG))
seg->charset= &my_charset_bin;
else
seg->charset= field->charset_for_protocol();
if (field->null_ptr)
{
seg->null_bit= field->null_bit;
seg->null_pos= (uint) (field->null_ptr - (uchar*) table_arg->record[0]);
}
else
{
seg->null_bit= 0;
seg->null_pos= 0;
}
if (field->flags & AUTO_INCREMENT_FLAG &&
table_arg->found_next_number_field &&
key == share->next_number_index)
{
/*
Store key number and type for found auto_increment key
We have to store type as seg->type can differ from it
*/
auto_key= key+ 1;
auto_key_type= field->key_type();
}
if (seg->type == HA_KEYTYPE_BIT)
{
seg->bit_length= ((Field_bit *) field)->bit_len;
seg->bit_start= ((Field_bit *) field)->bit_ofs;
seg->bit_pos= (uint) (((Field_bit *) field)->bit_ptr -
(uchar*) table_arg->record[0]);
}
else
{
seg->bit_length= seg->bit_start= 0;
seg->bit_pos= 0;
}
}
}
mem_per_row+= MY_ALIGN(share->reclength + 1, sizeof(char*));
if (table_arg->found_next_number_field)
{
keydef[share->next_number_index].flag|= HA_AUTO_KEY;
found_real_auto_increment= share->next_number_key_offset == 0;
}
hp_create_info->auto_key= auto_key;
hp_create_info->auto_key_type= auto_key_type;
hp_create_info->max_table_size=current_thd->variables.max_heap_table_size;
hp_create_info->with_auto_increment= found_real_auto_increment;
hp_create_info->internal_table= internal_table;
max_rows= (ha_rows) (hp_create_info->max_table_size / mem_per_row);
if (share->max_rows && share->max_rows < max_rows)
max_rows= share->max_rows;
hp_create_info->max_records= (ulong) max_rows;
hp_create_info->min_records= (ulong) share->min_rows;
hp_create_info->keys= share->keys;
hp_create_info->reclength= share->reclength;
hp_create_info->keydef= keydef;
return 0;
}
int ha_heap::create(const char *name, TABLE *table_arg,
HA_CREATE_INFO *create_info)
{
int error;
my_bool created;
HP_CREATE_INFO hp_create_info;
error= heap_prepare_hp_create_info(table_arg, internal_table,
&hp_create_info);
if (error)
return error;
hp_create_info.auto_increment= (create_info->auto_increment_value ?
create_info->auto_increment_value - 1 : 0);
error= heap_create(name, &hp_create_info, &internal_share, &created);
my_free(hp_create_info.keydef);
DBUG_ASSERT(file == 0);
return (error);
}
void ha_heap::update_create_info(HA_CREATE_INFO *create_info)
{
table->file->info(HA_STATUS_AUTO);
if (!(create_info->used_fields & HA_CREATE_USED_AUTO))
create_info->auto_increment_value= stats.auto_increment_value;
}
void ha_heap::get_auto_increment(ulonglong offset, ulonglong increment,
ulonglong nb_desired_values,
ulonglong *first_value,
ulonglong *nb_reserved_values)
{
ha_heap::info(HA_STATUS_AUTO);
*first_value= stats.auto_increment_value;
/* such table has only table-level locking so reserves up to +inf */
*nb_reserved_values= ULONGLONG_MAX;
}
bool ha_heap::check_if_incompatible_data(HA_CREATE_INFO *info,
uint table_changes)
{
/* Check that auto_increment value was not changed */
if ((info->used_fields & HA_CREATE_USED_AUTO &&
info->auto_increment_value != 0) ||
table_changes == IS_EQUAL_NO ||
table_changes & IS_EQUAL_PACK_LENGTH) // Not implemented yet
return COMPATIBLE_DATA_NO;
return COMPATIBLE_DATA_YES;
}
struct st_mysql_storage_engine heap_storage_engine=
{ MYSQL_HANDLERTON_INTERFACE_VERSION };
mysql_declare_plugin(heap)
{
MYSQL_STORAGE_ENGINE_PLUGIN,
&heap_storage_engine,
"MEMORY",
"MySQL AB",
"Hash based, stored in memory, useful for temporary tables",
PLUGIN_LICENSE_GPL,
heap_init,
NULL,
0x0100, /* 1.0 */
NULL, /* status variables */
NULL, /* system variables */
NULL, /* config options */
0, /* flags */
}
mysql_declare_plugin_end;
maria_declare_plugin(heap)
{
MYSQL_STORAGE_ENGINE_PLUGIN,
&heap_storage_engine,
"MEMORY",
"MySQL AB",
"Hash based, stored in memory, useful for temporary tables",
PLUGIN_LICENSE_GPL,
heap_init,
NULL,
0x0100, /* 1.0 */
NULL, /* status variables */
NULL, /* system variables */
"1.0", /* string version */
MariaDB_PLUGIN_MATURITY_STABLE /* maturity */
}
maria_declare_plugin_end;