Handle string length as size_t, consistently (almost always:))
Change function prototypes to accept size_t, where in the past
ulong or uint were used. change local/member variables to size_t
when appropriate.
This fix excludes rocksdb, spider,spider, sphinx and connect for now.
MDL_CONTEXT::TRY_ACQUIRE_LOCK_IMPL
ANALYSIS:
=========
Server sometimes exited when multiple threads tried to
acquire and release metadata locks simultaneously (for
example, necessary to access a table). The same problem
could have occurred when new objects were registered/
deregistered in Performance Schema.
The problem was caused by a bug in LF_HASH - our lock free
hash implementation which is used by metadata locking
subsystem in 5.7 branch. In 5.5 and 5.6 we only use LF_HASH
in Performance Schema Instrumentation implementation. So
for these versions, the problem was limited to P_S.
The problem was in my_lfind() function, which searches for
the specific hash element by going through the elements
list. During this search it loads information about element
checked such as key pointer and hash value into local
variables. Then it confirms that they are not corrupted by
concurrent delete operation (which will set pointer to 0)
by checking if element is still in the list. The latter
check did not take into account that compiler (and
processor) can reorder reads in such a way that load of key
pointer will happen after it, making result of the check
invalid.
FIX:
====
This patch fixes the problem by ensuring that no such
reordering can take place. This is achieved by using
my_atomic_loadptr() which contains compiler and processor
memory barriers for the check mentioned above and other
similar places.
The default (for non-Windows systems) implementation of
my_atomic*() relies on old __sync intrisics and implements
my_atomic_loadptr() as read-modify operation. To avoid
scalability/performance penalty associated with addition of
my_atomic_loadptr()'s we change the my_atomic*() to use
newer __atomic intrisics when available. This new default
implementation doesn't have such a drawback.
cherry-pick f1daf9ce from 10.0 branch
-------------------------------------
Fix build failures caused by new C runtime library
- isnan, snprintf, struct timespec are now defined, attempt to
redefine them leads
- P_tmpdir, tzname are no more defined
- lfind() and lsearch() in lf_hash.c had to be renamed, declaration
conflicts with some C runtime functions with the same name declared in
a header included by stdlib.h
Also fix couple of annoying warnings :
- remove #define NOMINMAX from config.h to avoid "redefined" compiler
warnings(NOMINMAX is already in compile flags)
- disable incremental linker in Debug as well (feature not used much
and compiler crashes often)
Also simplify package building with Wix, require Wix 3.9 or later
(VS2015 is not compatible with old Wix 3.5/3.6)
Problem Statement
=========
Fix various issues when building MySQL with Visual Studio 2015.
Fix:
=======
- Visual Studio 2015 adds support for timespec. Add check and
related code to use this and only use our replacement if
timespec is not defined.
- Rename lfind/lsearch to my* to avoid redefinition problems.
- Set default value for TMPDIR to "" on Windows as P_tmpdir
no longer exists.
- using VS definition of snprintf if available
- tzname are now renamed to _tzname.
Fix build failures caused by new C runtime library
- isnan, snprintf, struct timespec are now defined, attempt to
redefine them leads
- P_tmpdir, tzname are no more defined
- lfind() and lsearch() in lf_hash.c had to be renamed, declaration
conflicts with some C runtime functions with the same name declared in
a header included by stdlib.h
Also fix couple of annoying warnings :
- remove #define NOMINMAX from config.h to avoid "redefined" compiler
warnings(NOMINMAX is already in compile flags)
- disable incremental linker in Debug as well (feature not used much
and compiler crashes often)
Also simplify package building with Wix, require Wix 3.9 or later
(VS2015 is not compatible with old Wix 3.5/3.6)
if lf_hash_delete() and lf_hash_search() cannot create a new bucket
because of OOM, they'll start the search from the parent bucket.
As for lf_hash_iterate() - it only ever uses bucket number 0, so if it
cannot create *that* bucket, the hash must surely be empty.
casts, etc
real changes are:
* remove one retry, it is enough to check for DELETED
after the key is read
* advance 'head' pointer when we see a dummy node to have
shorter retries
A bug in the lock-free hash implementation!
when lsearch() has not found the key, the caller needs to unpin all the three pins,
because lsearch() was using all the three.
- leaking mutex in lf_hash_destroy
- pthread_getspecific() before pthread_key_create() in my_thread_var_dbug()
(called by static C++ object constructors called in sys_vars)
- perfschema destroys mutexes that were not created.
Essentially, the problem is that safemalloc is excruciatingly
slow as it checks all allocated blocks for overrun at each
memory management primitive, yielding a almost exponential
slowdown for the memory management functions (malloc, realloc,
free). The overrun check basically consists of verifying some
bytes of a block for certain magic keys, which catches some
simple forms of overrun. Another minor problem is violation
of aliasing rules and that its own internal list of blocks
is prone to corruption.
Another issue with safemalloc is rather the maintenance cost
as the tool has a significant impact on the server code.
Given the magnitude of memory debuggers available nowadays,
especially those that are provided with the platform malloc
implementation, maintenance of a in-house and largely obsolete
memory debugger becomes a burden that is not worth the effort
due to its slowness and lack of support for detecting more
common forms of heap corruption.
Since there are third-party tools that can provide the same
functionality at a lower or comparable performance cost, the
solution is to simply remove safemalloc. Third-party tools
can provide the same functionality at a lower or comparable
performance cost.
The removal of safemalloc also allows a simplification of the
malloc wrappers, removing quite a bit of kludge: redefinition
of my_malloc, my_free and the removal of the unused second
argument of my_free. Since free() always check whether the
supplied pointer is null, redudant checks are also removed.
Also, this patch adds unit testing for my_malloc and moves
my_realloc implementation into the same file as the other
memory allocation primitives.
client/mysqldump.c:
Pass my_free directly as its signature is compatible with the
callback type -- which wasn't the case for free_table_ent.
Essentially, the problem is that safemalloc is excruciatingly
slow as it checks all allocated blocks for overrun at each
memory management primitive, yielding a almost exponential
slowdown for the memory management functions (malloc, realloc,
free). The overrun check basically consists of verifying some
bytes of a block for certain magic keys, which catches some
simple forms of overrun. Another minor problem is violation
of aliasing rules and that its own internal list of blocks
is prone to corruption.
Another issue with safemalloc is rather the maintenance cost
as the tool has a significant impact on the server code.
Given the magnitude of memory debuggers available nowadays,
especially those that are provided with the platform malloc
implementation, maintenance of a in-house and largely obsolete
memory debugger becomes a burden that is not worth the effort
due to its slowness and lack of support for detecting more
common forms of heap corruption.
Since there are third-party tools that can provide the same
functionality at a lower or comparable performance cost, the
solution is to simply remove safemalloc. Third-party tools
can provide the same functionality at a lower or comparable
performance cost.
The removal of safemalloc also allows a simplification of the
malloc wrappers, removing quite a bit of kludge: redefinition
of my_malloc, my_free and the removal of the unused second
argument of my_free. Since free() always check whether the
supplied pointer is null, redudant checks are also removed.
Also, this patch adds unit testing for my_malloc and moves
my_realloc implementation into the same file as the other
memory allocation primitives.
Don't use static link by default (in compile-pentium) as some new systems doesn't have all static libraries available
Change type for functions in plugin.h:str_mysql_ftparser_param() to const unsigned char and string lengths to size_t.
One effect of the above change is that one needs to include mysql_global.h or define size_t before including plugin.h
This fixes a case where mysql_client_test failed with newer gcc that enables strict-aliasing by default
BUILD/compile-pentium:
Don't use static link by default as some new systems doesn't have all static libraries available
client/mysql_upgrade.c:
Remove not used variable
cmd-line-utils/readline/config_readline.h:
Define some constants to get rid of compiler warnings on Linux
cmd-line-utils/readline/display.c:
Get rid of compiler warnings
cmd-line-utils/readline/history.c:
Got rid of compiler warnings:
- Defining some strings as const
- Added cast
cmd-line-utils/readline/rlmbutil.h:
Added cast to get rid of compiler warnings
cmd-line-utils/readline/text.c:
Remove not needed initialization to get rid of compiler warnings
cmd-line-utils/readline/xmalloc.c:
Changed types to 'const char* to get rid of compiler warnings
configure.in:
Ensure that we use MariaDB as suffix
include/mysql/plugin.h:
Changed types to 'const unsigned char* to get rid of compiler warnings (in other parts of the code)
Change length for not \0 terminated string to size_t
include/mysql/plugin.h.pp:
Update related to plugin.h
libmysql/libmysql.c:
Fixed bug that caused core dump with newer gcc when strict aliasing is not turned off
mysql-test/t/information_schema.test:
Test is depending on innodb
mysql-test/t/not_partition.test:
Fixed wrong directory name
(Not noticed before as we don't ususally run this test)
mysys/lf_hash.c:
Got rid of compiler warnings from -Wstrict-aliasing
mysys/my_redel.c:
Removed not used variable
regex/engine.c:
Changed types to 'const char* to get rid of compiler warnings
regex/engine.ih:
Changed types to 'const char* to get rid of compiler warnings
sql/sp_head.cc:
Got rid of compiler warning from -Wstrict-aliasing
sql/sql_base.cc:
Got rid of compiler warnings from -Wstrict-aliasing
(The original code was probably wrong as nj_col->table_field was
sql/sql_builtin.cc.in:
plugin.h needs to have size_t defined
sql/sql_parse.cc:
Remove used variable
sql/sql_select.cc:
Got rid of compiler warnings from -Wstrict-aliasing
sql/sql_show.cc:
Added #ifdef to get rid of compiler warning when not using partition engine
sql/table.cc:
Got rid of compiler warning from -Wstrict-aliasing
storage/maria/ha_maria.cc:
Got rid of compiler warnings from -Wstrict-aliasing:
- Use the thd_killed() API function
storage/maria/lockman.c:
Got rid of compiler warnings from -Wstrict-aliasing
storage/maria/ma_check.c:
Got rid of compiler warnings from -Wstrict-aliasing
Change to use new version of _ma_killed_ptr; Don't call it as often as before
storage/maria/ma_check_standalone.h:
Update to compatible _ma_killed_ptr() from ha_maria.cc
storage/maria/ma_ft_boolean_search.c:
Changed pointers from char -> const char* and length to size_t (to get rid of compiler warnings and casts)
storage/maria/ma_ft_nlq_search.c:
Got rid of compiler warnings from -Wstrict-aliasing
Ensure that 'subkeys' is 32 bit
storage/maria/ma_ft_parser.c:
Changed pointers from char -> const char* and length to size_t (to get rid of compiler warnings and casts)
storage/maria/ma_ftdefs.h:
Changed pointers from char -> const char* and length to size_t (to get rid of compiler warnings and casts)
storage/maria/ma_sort.c:
Change to use new version of _ma_killed_ptr; Don't call it as often as before
storage/maria/ma_state.c:
Got rid of compiler warnings from -Wstrict-aliasing
storage/maria/maria_def.h:
Redefine ma_killed_ptr()
storage/maria/maria_ftdump.c:
Got rid of compiler warnings from -Wstrict-aliasing
storage/maria/trnman.c:
Got rid of compiler warnings from -Wstrict-aliasing
storage/myisam/ft_boolean_search.c:
Changed pointers from char -> const char* and length to size_t (to get rid of compiler warnings and casts)
storage/myisam/ft_nlq_search.c:
Got rid of compiler warnings from -Wstrict-aliasing
storage/myisam/ft_parser.c:
Changed pointers from char -> const char* and length to size_t (to get rid of compiler warnings and casts)
storage/myisam/ft_stopwords.c:
Changed pointers from char -> const char* and length to size_t (to get rid of compiler warnings and casts)
storage/myisam/ftdefs.h:
Changed pointers from char -> const char* and length to size_t (to get rid of compiler warnings and casts)
storage/myisam/ha_myisam.cc:
Got rid of compiler warnings from -Wstrict-aliasing:
- Use the thd_killed() API function
storage/myisam/mi_check.c:
Use new killed_ptr() function
storage/myisam/myisam_ftdump.c:
Got rid of compiler warnings from -Wstrict-aliasing
storage/myisam/myisamchk.c:
Update to compatible killed_ptr() from ha_myisam.cc
storage/myisam/myisamdef.h:
Redefine killed_ptr()
storage/myisam/myisamlog.c:
Got rid of compiler warnings from -Wstrict-aliasing
storage/myisam/sort.c:
Change to use new version of killed_ptr; Don't call it as often as before
storage/xtradb/fil/fil0fil.c:
Fixedc ompiler warning
storage/xtradb/trx/trx0i_s.c:
Include mysql_plugin.h later to ensure that size_t is defined
include/atomic/generic-msvc.h:
prevent possible compiler warnings
include/lf.h:
comments, better definition for LF_HASH_OVERHEAD
include/maria.h:
define MARIA_CANNOT_ROLLBACK here
include/my_pthread.h:
avoid possible name clash
include/waiting_threads.h:
comments, const, move WT_RESOURCE to waiting_threads.c
mysql-test/suite/maria/r/maria_notembedded.result:
new test
mysql-test/suite/maria/t/maria_notembedded.test:
new test - 5-way deadlock
mysys/lf_hash.c:
better definition for LF_HASH_OVERHEAD
mysys/my_static.c:
comment
mysys/my_thr_init.c:
casts
mysys/waiting_threads.c:
comments, asserts, etc
server-tools/instance-manager/parse.cc:
fix my_init_dynamic_array() to follow new calling conventions
sql/mysqld.cc:
call wt_init after set_proper_floating_point_mode
sql/sql_class.h:
comment
storage/maria/ha_maria.cc:
move MARIA_CANNOT_ROLLBACK to a common header
storage/maria/ma_commit.c:
comment
storage/maria/ma_write.c:
comments, check for HA_ERR_FOUND_DUPP_KEY
storage/maria/trnman.c:
comments, assert
storage/maria/trnman.h:
comments
storage/maria/unittest/trnman-t.c:
be paranoid
unittest/mysys/lf-t.c:
comments
unittest/mysys/waiting_threads-t.c:
comments, safety, memory leak
include/waiting_threads.h:
don't #pragma pack
mysys/lf_hash.c:
typo in a comment
mysys/waiting_threads.c:
use the size of data, not the size of (possibly padded) structure
client/mysqltest.c:
compiler warnings
configure.in:
remove old tests for unused programs
disable the use of gcc built-ins if smp assembler atomics were selected explictily.
add waiting_threads.o to THREAD_LOBJECTS
include/lf.h:
replace the end-of-stack pointer with the pointer to the end-of-stack pointer.
the latter could be stored in THD (mysys_vars) and updated in pool-of-threads
scheduler.
constructor/destructor in lf-alloc
include/my_pthread.h:
shuffle set_timespec/set_timespec_nsec macros a bit to be able to fill
several timeout structures with only one my_getsystime() call
include/waiting_threads.h:
waiting threads - wait-for graph and deadlock detection
mysys/Makefile.am:
add waiting_threads.c
mysys/lf_alloc-pin.c:
replace the end-of-stack pointer with the pointer to the end-of-stack pointer.
the latter could be stored in THD (mysys_vars) and updated in pool-of-threads
scheduler.
constructor/destructor in lf-alloc
mysys/lf_hash.c:
constructor/destructor in lf-alloc
mysys/my_thr_init.c:
remember end-of-stack pointer in the mysys_var
mysys/waiting_threads.c:
waiting threads - wait-for graph and deadlock detection
storage/maria/ha_maria.cc:
replace the end-of-stack pointer with the pointer to the end-of-stack pointer.
the latter could be stored in THD (mysys_vars) and updated in pool-of-threads
scheduler.
storage/maria/ma_commit.c:
replace the end-of-stack pointer with the pointer to the end-of-stack pointer.
the latter could be stored in THD (mysys_vars) and updated in pool-of-threads
scheduler.
storage/maria/trnman.c:
replace the end-of-stack pointer with the pointer to the end-of-stack pointer.
the latter could be stored in THD (mysys_vars) and updated in pool-of-threads
scheduler.
storage/maria/trnman_public.h:
replace the end-of-stack pointer with the pointer to the end-of-stack pointer.
the latter could be stored in THD (mysys_vars) and updated in pool-of-threads
scheduler.
storage/maria/unittest/trnman-t.c:
replace the end-of-stack pointer with the pointer to the end-of-stack pointer.
the latter could be stored in THD (mysys_vars) and updated in pool-of-threads
scheduler.
unittest/mysys/Makefile.am:
add waiting_threads-t
unittest/mysys/lf-t.c:
factor out the common code for multi-threaded stress unit tests
move lf tests to a separate file
unittest/mysys/my_atomic-t.c:
factor out the common code for multi-threaded stress unit tests
move lf tests to a separate file
unittest/mysys/thr_template.c:
factor out the common code for multi-threaded stress unit tests
unittest/mysys/waiting_threads-t.c:
wt tests
