Problem: For every event read, mysqlbinlog calls localtime() which in turn
calls stat(/etc/localtime) which is causing kernel mutex contention.
Analysis and Fix:
localtime() calls stat(/etc/localtime) for every instance of the call
where as localtime_r() the reentrant version was optimized to store
the read only tz internal structure. Hence it will not call
stat(/etc/localtime). It will call only once at the beginning.
The mysql server is calling localtime_r() and mysqlbinlog tool is
one place where we are still using localtime().
Once the process (mysqlbinlog) is started if timezone is changed
it will be not picked up the the process and it will continue
with the same values as the beginning of the process. This
behavior is in-lined with mysql server.
Also adding localtime_r() and gmtime_r() support for windows.
- Added --verbose to BUILD scripts to get make to write out compile commands.
- Detect if AM_EXTRA_MAKEFLAGS=VERBOSE=1 was used with build scripts.
- Don't write warnings about replication variables when doing bootstrap.
- Fixed that mysql_cond_wait() and mysql_cond_timedwait() will report original source file in case of errors.
- Ignore some compiler warnings
BUILD/FINISH.sh:
Detect if AM_EXTRA_MAKEFLAGS=VERBOSE=1 or --verbose was used
BUILD/SETUP.sh:
Added --verbose to print out the full compile lines
Updated help message
client/mysqltest.cc:
Fixed that one can use 'replace' with cat_file
cmake/configure.pl:
If --verbose is used, get make to write out compile commands
debian/dist/Debian/rules:
Added $AM_EXTRA_MAKEFLAGS to get VERBOSE=1 on buildbot builds
debian/dist/Ubuntu/rules:
Added $AM_EXTRA_MAKEFLAGS to get VERBOSE=1 on buildbot builds
include/my_pthread.h:
Made set_timespec_time_nsec() more portable.
include/mysql/psi/mysql_thread.h:
Fixed that mysql_cond_wait() and mysql_cond_timedwait() will report original source file in case of errors.
mysql-test/suite/innodb/r/auto_increment_dup.result:
Fixed wrong DBUG_SYNC
mysql-test/suite/innodb/t/auto_increment_dup.test:
Fixed wrong DBUG_SYNC
mysql-test/suite/perfschema/include/upgrade_check.inc:
Make test more portable for changes in *.sql files
mysql-test/suite/perfschema/r/pfs_upgrade.result:
Updated test results
mysql-test/valgrind.supp:
Ignore running Aria checkpoint thread
scripts/mysqlaccess.sh:
Changed reference of bugs database
Ensure that also client-server group is read.
sql/handler.cc:
Added missing syncpoint
sql/mysqld.cc:
Don't write warnings about replication variables when doing bootstrap
sql/mysqld.h:
Don't write warnings about replication variables when doing bootstrap
sql/rpl_rli.cc:
Don't write warnings about replication variables when doing bootstrap
sql/sql_insert.cc:
Don't mask SERVER_SHUTDOWN in insert_delayed
This is done to be able to distingush between shutdown and interrupt errors
support-files/compiler_warnings.supp:
Ignore some compiler warnings in xtradb,innobase, oqgraph, yassl, string3.h
sql/sql_insert.cc:
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
******
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
sql/sql_table.cc:
small cleanup
******
small cleanup
A lot of small fixes and new test cases.
client/mysqlbinlog.cc:
Cast removed
client/mysqltest.cc:
Added missing DBUG_RETURN
include/my_pthread.h:
set_timespec_time_nsec() now only takes one argument
mysql-test/t/date_formats.test:
Remove --disable_ps_protocl as now also ps supports microseconds
mysys/my_uuid.c:
Changed to use my_interval_timer() instead of my_getsystime()
mysys/waiting_threads.c:
Changed to use my_hrtime()
sql/field.h:
Added bool special_const_compare() for fields that may convert values before compare (like year)
sql/field_conv.cc:
Added test to get optimal copying of identical temporal values.
sql/item.cc:
Return that item_int is equal if it's positive, even if unsigned flag is different.
Fixed Item_cache_str::save_in_field() to have identical null check as other similar functions
Added proper NULL check to Item_cache_int::save_in_field()
sql/item_cmpfunc.cc:
Don't call convert_constant_item() if there is nothing that is worth converting.
Simplified test when years should be converted
sql/item_sum.cc:
Mark cache values in Item_sum_hybrid as not constants to ensure they are not replaced by other cache values in compare_datetime()
sql/item_timefunc.cc:
Changed sec_to_time() to take a my_decimal argument to ensure we don't loose any sub seconds.
Added Item_temporal_func::get_time() (This simplifies some things)
sql/mysql_priv.h:
Added Lazy_string_decimal()
sql/mysqld.cc:
Added my_decimal constants max_seconds_for_time_type, time_second_part_factor
sql/table.cc:
Changed expr_arena to be of type CONVENTIONAL_EXECUTION to ensure that we don't loose any items that are created by fix_fields()
sql/tztime.cc:
TIME_to_gmt_sec() now sets *in_dst_time_gap in case of errors
This is needed to be able to detect if timestamp is 0
storage/maria/lockman.c:
Changed from my_getsystime() to set_timespec_time_nsec()
storage/maria/ma_loghandler.c:
Changed from my_getsystime() to my_hrtime()
storage/maria/ma_recovery.c:
Changed from my_getsystime() to mmicrosecond_interval_timer()
storage/maria/unittest/trnman-t.c:
Changed from my_getsystime() to mmicrosecond_interval_timer()
storage/xtradb/handler/ha_innodb.cc:
Added support for new time,datetime and timestamp
unittest/mysys/thr_template.c:
my_getsystime() -> my_interval_timer()
unittest/mysys/waiting_threads-t.c:
my_getsystime() -> my_interval_timer()
* my_getsystime() is only an interval timer. Its value can beused for calculating
time intervals.
* renamed my_getsystime() to my_interval_timer(), to make the semantics
clearer and let the compiler catch wrong usages of my_getsystime()
(also future ones, that may come in merges).
* increased its granularity from 100ns to 1ns, old value was for UUID,
but as UUID can no longer use it directly there is no need to downgrade
the OS provided value
* fixed the UUID code to anchor the my_interval_timer() on the epoch, as
required by the UUID standard. That is, this was only needed by UUID,
and now I've moved it to UUID code from my_getsystime().
* fixed other wrong usages of my_getsystime() - e.g. in calculating
times for pthread_cond_timedwait. It was buggy and could've caused
long waits if OS clock would be changed.
Bug#11763065 - 55730: KILL_SERVER() CALLS SETEVENT ON A NULL
HANDLE, SMEM_EVENT_CONNECT_REQUEST
Application Verifier is a Microsoft tool used for
detecting certain classes of programming errors.
In particular, MS Windows OS resource usage is
monitored for wrong usage (handles, thread local
storage, critical sections, ...)
In MySQL 5.5.x, an error was introduced where an
object on thread local storage was used before the
TLS and the object was created.
The fix has been to move the mysys initialization
to an earlier stage in the boot process when built for
Windows. For non-win builds, the init already happens
early.
Some un-tangling of calls to my_init(), my_basic_init()
and my_thread_global_init() was done. There is no
longer a need to do init in steps, so the full my_init()
is called instead of my_init_basic().
In addition, Bug#11763065 was fixed. The event handle
'smem_event_connect_request' is only created if
'opt_enable_shared_memory' is set. When killing the
server, an event was flagged on the handle
unconditionally. Added a test, so it will only be
flagged if created.
include/my_pthread.h:
my_thread_basic_global_init is no longer
necessary, and the my_thread_basic_global_reinit
function is renamed to reflect that it now
reinits mutexes and condvars originating from
my_thread_global_init
mysys/my_thr_init.c:
Reorganized code.
- Fixed some issues with partitions and connection_string, which also fixed lp:716890 "Pre- and post-recovery crash in Aria"
- Fixed wrong assert in Aria
Now need to merge with latest xtradb before pushing
sql/ha_partition.cc:
Ensure that m_ordered_rec_buffer is not freed before close.
sql/mysqld.cc:
Changed to use opt_stack_trace instead of opt_pstack.
Removed references to pstack
sql/partition_element.h:
Ensure that connect_string is initialized
storage/maria/ma_key_recover.c:
Fixed wrong assert
configure.in:
* don't use assembler when building with native linker and --with-embedded.
Embedded uses PIC and our assembler sources aren't PIC
* workaround for OpenSolaris Bug 6611808
extra/libevent/devpoll.c:
compiler warning
extra/yassl/src/buffer.cpp:
include config.h first to make sure that _FILE_OFFSET_BITS is defined
before including system headers
extra/yassl/taocrypt/benchmark/benchmark.cpp:
include config.h first to make sure that _FILE_OFFSET_BITS is defined
before including system headers
extra/yassl/taocrypt/test/test.cpp:
include config.h first to make sure that _FILE_OFFSET_BITS is defined
before including system headers
include/my_pthread.h:
workaround for OpenSolaris Bug 6611808
and 'THREAD_SAFE_CLIENT'.
As of MySQL 5.5, we no longer support non-threaded
builds. This patch removes all references to the
obsolete THREAD and THREAD_SAFE_CLIENT preprocessor
symbols. These were used to distinguish between
threaded and non-threaded builds.
cmake/os/SunOS.cmake:
Remove TARGET_OS_SOLARIS
config.h.cmake:
Remove TARGET_OS_SOLARIS
Add PTHREAD_ONCE_INITIALIZER
configure.cmake:
Add function for testing whether we need { PTHREAD_ONCE_INIT } rather than PTHREAD_ONCE_INIT
include/my_pthread.h:
Use PTHREAD_ONCE_INITIALIZER if set by cmake.
include/mysql/psi/mysql_file.h:
Include my_global.h first, to get correct platform definitions.
mysys/ptr_cmp.c:
Hide the unused static functions in #ifdef's on solaris.
Use __sun (defined by both gcc and SunPro cc) rather than TARGET_OS_SOLARIS
sql/my_decimal.cc:
Include my_global.h first, to get correct platform definitions.
sql/mysqld.cc:
Fix signed/unsigned comparison warning.
sql/sql_audit.h:
Include my_global.h first, to get correct platform definitions.
sql/sql_plugin.h:
Include my_global.h first, to get correct platform definitions.
sql/sql_show.cc:
Fix: warning: cast from pointer to integer of different size
sql/sys_vars.h:
Use reinterpret_cast rather than c-style cast.
storage/perfschema/pfs_instr.cc:
Include my_global.h first, to get correct platform definitions.
- Remove all mentioning of /MAP /MAPINFO link options (does not work in VS2010).
Remove map files from packaging.
- Fix warning about ETIMEDOUT being redefined.
- Fix warning about FSP_EXTENT_SIZE in xtradb (32/64 bit right shift mismatch)
- Silence warnings coming from generated code (flex/bison) in xtradb/innodb_plugin.
- Be nice to people without cygwin (me) and add win/configure-mariadb.bat with options suitable for quick compilation, e.g no embedded
Open issues:
- A better fix for #57688; Igor is working on this
- Test failure in index_merge_innodb.test ; Igor promised to look at this
- Some Innodb tests fails (need to merge with latest xtradb) ; Kristian promised to look at this.
- Failing tests: innodb_plugin.innodb_bug56143 innodb_plugin.innodb_bug56632 innodb_plugin.innodb_bug56680 innodb_plugin.innodb_bug57255
- Werror is disabled; Should be enabled after merge with xtradb.
Fix assorted compiler warnings.
include/my_pthread.h:
Like for pthread_cond_timedwait, the abstime is constant.
mysys/my_gethwaddr.c:
Instead of using a manual copy that introduce warnings due to
type mismatch, copy the buffer using memcpy and use memcmp to
check whether all bytes of the buffer are zeroed.
mysys/thr_mutex.c:
Like for pthread_cond_timedwait, the abstime is constant.
unittest/mytap/tap.h:
Introduce a ok() variant that does not take a format argument.
Since ok() is tagged with a printf attribute, GCC complains if
the fmt argument is NULL.
detector". This patch addresses performance regression in OLTP_RO/MyISAM
test on Windows introduced by the fix for bug #56405. Thus it makes
original patch acceptable as a solution for bug #56585 "Slowdown of
readonly sysbench benchmarks (e.g point_select) on Windows 5.5".
With this patch, MySQL will use native Windows condition variables and
reader-writer locks if they are supported by the OS.
This speeds up MyISAM and the effect comes mostly from using native
rwlocks. Native conditions improve scalability with higher number of
concurrent users in other situations, e.g for prlocks.
Benchmark numbers for this patch as measured on Win2008R2 quad
core machine are attached to the bug report.
( direct link http://bugs.mysql.com/file.php?id=15883 )
Note that currently we require at least Windows7/WS2008R2 for
reader-writer locks, even though native rwlock is available also on Vista.
Reason is that "trylock" APIs are missing on Vista, and trylock is used in
the server (in a single place in query cache).
While this patch could have been written differently, to enable the native
rwlock optimization also on Vista/WS2008 (e.g using native locks everywhere
but portable implementation in query cache), this would come at the
expense of the code clarity, as it would introduce a new "try-able" rwlock
type, to handle Vista case.
Another way to improve performance for the special case
(OLTP_RO/MYISAM/Vista) would be to eliminate "trylock" usage from server,
but this is outside of the scope here.
Native conditions variables are used beginning with Vista though the effect
of using condition variables alone is not measurable in this benchmark.
But when used together with native rwlocks on Win7, native conditions improve
performance in high-concurrency OLTP_RO/MyISAM (128 and more sysbench
users).
detector" that doesn't introduce bug #56715 "Concurrent
transactions + FLUSH result in sporadical unwarranted
deadlock errors".
Deadlock could have occurred when workload containing a mix
of DML, DDL and FLUSH TABLES statements affecting the same
set of tables was executed in a heavily concurrent environment.
This deadlock occurred when several connections tried to
perform deadlock detection in the metadata locking subsystem.
The first connection started traversing wait-for graph,
encountered a sub-graph representing a wait for flush, acquired
LOCK_open and dived into sub-graph inspection. Then it
encountered sub-graph corresponding to wait for metadata lock
and blocked while trying to acquire a rd-lock on
MDL_lock::m_rwlock, since some,other thread had a wr-lock on it.
When this wr-lock was released it could have happened (if there
was another pending wr-lock against this rwlock) that the rd-lock
from the first connection was left unsatisfied but at the same
time the new rd-lock request from the second connection sneaked
in and was satisfied (for this to be possible the second
rd-request should come exactly after the wr-lock is released but
before pending the wr-lock manages to grab rwlock, which is
possible both on Linux and in our own rwlock implementation).
If this second connection continued traversing the wait-for graph
and encountered a sub-graph representing a wait for flush it tried
to acquire LOCK_open and thus the deadlock was created.
The previous patch tried to workaround this problem by not
allowing the deadlock detector to lock LOCK_open mutex if
some other thread doing deadlock detection already owns it
and current search depth is greater than 0. Instead deadlock
was reported. As a result it has introduced bug #56715.
This patch solves this problem in a different way.
It introduces a new rw_pr_lock_t implementation to be used
by MDL subsystem instead of one based on Linux rwlocks or
our own rwlock implementation. This new implementation
never allows situation in which an rwlock is rd-locked and
there is a blocked pending rd-lock. Thus the situation which
has caused this bug becomes impossible with this implementation.
Due to fact that this implementation is optimized for
wr-lock/unlock scenario which is most common in the MDL
subsystem it doesn't introduce noticeable performance
regressions in sysbench tests. Moreover it significantly
improves situation for POINT_SELECT test when many
connections are used.
No test case is provided as this bug is very hard to repeat
in MTR environment but is repeatable with the help of RQG
tests.
This patch also doesn't include a test for bug #56715
"Concurrent transactions + FLUSH result in sporadical
unwarranted deadlock errors" as it takes too much time to
be run as part of normal test-suite runs.
config.h.cmake:
We no longer need to check for presence of
pthread_rwlockattr_setkind_np as we no longer
use Linux-specific implementation of rw_pr_lock_t
which uses this function.
configure.cmake:
We no longer need to check for presence of
pthread_rwlockattr_setkind_np as we no longer
use Linux-specific implementation of rw_pr_lock_t
which uses this function.
configure.in:
We no longer need to check for presence of
pthread_rwlockattr_setkind_np as we no longer
use Linux-specific implementation of rw_pr_lock_t
which uses this function.
include/my_pthread.h:
Introduced new implementation of rw_pr_lock_t.
Since it never allows situation in which rwlock is rd-locked
and there is a blocked pending rd-lock it is not affected by
bug #56405 "Deadlock in the MDL deadlock detector".
This implementation is also optimized for wr-lock/unlock
scenario which is most common in MDL subsystem. So it doesn't
introduce noticiable performance regressions in sysbench tests
(compared to old Linux-specific implementation). Moreover it
significantly improves situation for POINT_SELECT test when
many connections are used.
As part of this change removed try-lock part of API for
this type of lock. It is not used in our code and it would
be hard to implement correctly within constraints of new
implementation.
Finally, removed support of preferring readers from
my_rw_lock_t implementation as the only user of this
feature was old rw_pr_lock_t implementation.
include/mysql/psi/mysql_thread.h:
Removed try-lock part of prlock API.
It is not used in our code and it would be hard
to implement correctly within constraints of new
prlock implementation.
mysys/thr_rwlock.c:
Introduced new implementation of rw_pr_lock_t.
Since it never allows situation in which rwlock is rd-locked
and there is a blocked pending rd-lock it is not affected by
bug #56405 "Deadlock in the MDL deadlock detector".
This implementation is also optimized for wr-lock/unlock
scenario which is most common in MDL subsystem. So it doesn't
introduce noticiable performance regressions in sysbench tests
(compared to old Linux-specific implementation). Moreover it
significantly improves situation for POINT_SELECT test when
many connections are used.
Also removed support of preferring readers from
my_rw_lock_t implementation as the only user of this
feature was old rw_pr_lock_t implementation.
- Changed to still use bcmp() in certain cases becasue
- Faster for short unaligneed strings than memcmp()
- Bettern when using valgrind
- Changed to use my_sprintf() instead of sprintf() to get higher portability for old systems
- Changed code to use MariaDB version of select->skip_record()
- Removed -%::SCCS/s.% from Makefile.am:s to remove automake warnings
Implement a few simple asserts in my_rwlock_t locks.
include/my_pthread.h:
Declare two simple assert functions.
include/mysql/psi/mysql_thread.h:
Add wrappers for new assert functions.
mysys/thr_rwlock.c:
Add asserts.
sql/sql_base.cc:
Silence a compiler warning for the case when
SAFE_MUTEX is not ON.
Fix warnings flagged by the new warning option -Wunused-but-set-variable
that was added to GCC 4.6 and that is enabled by -Wunused and -Wall. The
option causes a warning whenever a local variable is assigned to but is
later unused. It also warns about meaningless pointer dereferences.
client/mysql.cc:
Meaningless pointer dereferences.
client/mysql_upgrade.c:
Check whether reading from the file succeeded.
extra/comp_err.c:
Unused.
extra/yassl/src/yassl_imp.cpp:
Skip instead of reading data that is discarded.
include/my_pthread.h:
Variable is only used in debug builds.
include/mysys_err.h:
Add new error messages.
mysys/errors.c:
Add new error message for permission related functions.
mysys/mf_iocache.c:
Variable is only checked under THREAD.
mysys/my_copy.c:
Raise a error if chmod or chown fails.
mysys/my_redel.c:
Raise a error if chmod or chown fails.
regex/engine.c:
Use a equivalent variable for the assert.
server-tools/instance-manager/instance_options.cc:
Unused.
sql/field.cc:
Unused.
sql/item.cc:
Unused.
sql/log.cc:
Do not ignore the return value of freopen: only set buffer if
reopening succeeds.
Adjust doxygen comment to the right function.
Pass message lenght to log function.
sql/mysqld.cc:
Do not ignore the return value of freopen: only set buffer if
reopening succeeds.
sql/partition_info.cc:
Unused.
sql/slave.cc:
No need to set pointer to the address of '\0'.
sql/spatial.cc:
Unused. Left for historical purposes.
sql/sql_acl.cc:
Unused.
sql/sql_base.cc:
Pointers are always set to the same variables.
sql/sql_parse.cc:
End statement if reading fails.
Store the buffer after it has actually been updated.
sql/sql_repl.cc:
No need to set pointer to the address of '\0'.
sql/sql_show.cc:
Put variable under the same ifdef block.
sql/udf_example.c:
Set null pointer flag appropriately.
storage/csv/ha_tina.cc:
Meaningless dereferences.
storage/example/ha_example.cc:
Return the error since it's available.
storage/myisam/mi_locking.c:
Remove unused and dead code.
Most of the changes are backports from MySQL 5.5. Please note
that the 64-bit build fails with VS 2010 and the 32-bit build
has problems running mysql-test-run.pl.
* Added files for compiling with VS 2010 and added them
to Makefile.am.
* ifdef'ed ETIMEDOUT, because it is defined by VS 2010 now
* Removed not needed /MAP's from cmake files
The atomic operations implementation on 5.1 has a few problems,
which might cause tests to abort randomly. Since no code in 5.1
uses atomic operations, simply remove the code.
This patch:
- Moves all definitions from the mysql_priv.h file into
header files for the component where the variable is
defined
- Creates header files if the component lacks one
- Eliminates all include directives from mysql_priv.h
- Eliminates all circular include cycles
- Rename time.cc to sql_time.cc
- Rename mysql_priv.h to sql_priv.h
on Windows".
On platforms where read-write lock implementation does not
prefer readers by default (Windows, Solaris) server might
have deadlocked while detecting MDL deadlock.
MDL deadlock detector relies on the fact that read-write
locks which are used in its implementation prefer readers
(see new comment for MDL_lock::m_rwlock for details).
So far MDL code assumed that default implementation of
read/write locks for the system has this property.
Indeed, this turned out ot be wrong, for example, for
Windows or Solaris. Thus MDL deadlock detector might have
deadlocked on these systems.
This fix simply adds portable implementation of read/write
lock which prefer readers and changes MDL code to use this
new type of synchronization primitive.
No test case is added as existing rqg_mdl_stability test can
serve as one.
config.h.cmake:
Check for presence of pthread_rwlockattr_setkind_np to be
able to determine if system natively supports read-write
locks for which we can specify if readers or writers should
be preferred.
configure.cmake:
Check for presence of pthread_rwlockattr_setkind_np to be
able to determine if system natively supports read-write
locks for which we can specify if readers or writers should
be preferred.
configure.in:
Check for presence of pthread_rwlockattr_setkind_np to be
able to determine if system natively supports read-write
locks for which we can specify if readers or writers should
be preferred.
include/my_pthread.h:
Added support for portable read-write locks which prefer
readers.
To do so extended existing my_rw_lock_t implementation to
support selection of whom to prefer depending on a flag.
mysys/thr_rwlock.c:
Extended existing my_rw_lock_t implementation to support
selection of whom to prefer depending on a flag.
Added rw_pr_init() function implementing initialization of
read-write locks preferring readers.
sql/mdl.cc:
Use portable read-write locks which prefer readers instead of
relying on that system implementation of read-write locks has
this property (this was true for Linux/NPTL but was false,
for example, for Windows and Solaris).
Added comment explaining why preferring readers is important
for MDL deadlock detector (thanks to Serg for example!).
sql/mdl.h:
Use portable read-write locks which prefer readers instead of
relying on that system implementation of read-write locks has
this property (this was true for Linux/NPTL but was false,
for example, for Windows and Solaris).
